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Abstract 

We give a new approach on general systems of the form 

, . J -A p u = - div(| Vu| p ~ 2 Vu) = £\ |a;| a u s v 5 , 
^ '{ -A q v = -div(|Vw| 9 ~ 2 Vu) = e 2 \x\ b u^v m , 

where Q,p,q,S,fi,s,m, a, & are real parameters, Q,p,q 7^ 1, and e\ = ±1, £2 = ±1- In the 
radial case we reduce the problem to a quadratic system of four coupled first order autonomous 
equations, of Kolmogorov type. It allows to obtain new local and global existence or nonexistence 
results. We consider in particular the case E\ — £2 = 1. We describe the behaviour of the ground 
states in two cases where the system is variational. We give a result of existence of ground states 
for a nonvariational system with p — q = 2 and s = m > 0, that improves the former ones. 
It is obtained by introducing a new type of energy function. In the nonradial case we solve 
a conjecture of nonexistence of ground states for the system with p = q = 2, 5 = m + 1 and 

[1 = 8+1. 
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1 Introduction 



In this paper we consider the nonnegative solutions of Emden-Fowler equations or systems in 
R N (N ^ 1), 

-div(|V-u| p_2 Vu) = e^ \x\ a u c 



(G) 



A p u = 

—A p u 
-A q v-- 



- div(|Vtt| p ~ 2 Vu) = ex \x\ a uV, 
div(\Vv\ q ~ 2 Vu) = e 2 \x\ h u^v m , 



£\ \X\ 
= E\ \x 



(1.1) 

(1.2) 



where Q,p, q, 5, \i, s, m, a, b are real parameters, Q,p,q ^ 1, and S\ = ±1, e 2 = ±1. These problems 
are the subject of a very rich litterature, either in the case of source terms {e\ =£2 = 1) or 
absorption terms {e\ =£2 = 1) or mixed terms (ei = —£2)- In the sequel we are concerned by the 
radial solutions, except at Section 9 where the solutions may be nonradial. 

In this article we we give a new way of studying the radial solutions. In Section [2] we reduce 
system (G) to a quadratic autonomous system: 



(mW 



X 
Y 



X 
Y - 



N-p , Z 
p—1 p— 1 



N-q 



q-l 

Z [N + a - sX 



+ 



w 

q-l 

8Y- 



Z], 



W t = W [N + b - ixX - mY - W] , 



where t = lnr, and 



ru 



rv 



X(t) = , Y(t) = , 

u v 



Z(t) 



-£ ir l+a U S V & , 



W{t) 



-e 2 r 1+6 «' 4 « Tn 1 



,/[<? • 



(1.3) 



This system is of Kolmogorov type. The reduction is valid for equations and systems with source 
terms , absorption terms , or mixed terms . It is remarkable that in the new system, p and q appear 
only as simple coefficients, which allows to treat any value of the parameters, even p or q < 1, and 
s, m, 6 or /i < 0. 

In Section [3] we revisit the well-known scalar case (jl.ip . where (G) becomes two-dimensional. We 
show that the phase plane of the system gives at the same time the behaviour of the two equations 

—A p u = |x| a n^ and 

which is a kind of unification of the two problems, with source terms or absorption terms. For the 
case of source term (ei = 1), we find again the results of [2], [19], showing that the new dynamical 
approach is simple and does not need regularity results or energy functions. Moreover it gives a 
model for the study of system (G). Indeed if p = q, a = b and 5 + s = + m, system (G) admits 
solutions of the form (u, u), where u is a solution of (jl.ip with Q = 5 + s. 

In the sequel of the article we study the case of source terms, i.e. (G) = (S), where 



A p u 



\x\ a U Q, 



(S) 



-A p u 
-A q v- 



\x\ a u s v 5 , 
\x\ b u^v m . 



(1.4) 



This system has been studied by many authors, in particular the Hamiltonian problem s = m = 0, 
in the linear case p = q = 2, see for example [20], [31], [29], [9], [33], [H], and the potential system 



2 



where 6 = m+1, [A = s + 1 and a = b, see [7J, [31], [35]; the problem with general powers has been 
studied in [3], [39], [3D], [UJ in the linear case and [6J, [12], [12] in the quasilinear case, see also [JJ, 

Here we suppose that 5, [i > 0, so that the system is always coupled, s,m ^ 0, and we assume 
for simplicity 

l<p, <? < iV, min(p + a, <? + 6) > 0, D = Sfi - (p - 1 - s)(g - 1 - m) > 0. (1.5) 

We say that a positive solution (u, v) in (0, i?) is regular at if u, v £ C 2 (0, i?)nC([0, /?)). Condition 
min(p+a, g+fr) > guaranties the existence of local regular solutions. Then u,v £ C 1 ([0, i?)). when 
a, 6 > —1, and u'(0) = v'(0) = 0. The assumption D > is a classical condition of superlinearity 
for the system. 

We are interessed in the existence or nonexistence of ground states, called G.S., that means 
global positive (u,v) in (0, oo) and regular at 0. We exclude the case of "trivial" solutions, (u,v) = 
(0, C) or (C, 0) , where C is a constant, which can exist when s > or m > 0. 

In Section HJ we give a series of local existence or nonexistence results concerning system (5), 
which complete the nonexistence results found in the litterature. They are not based on the fixed 
point method, quite hard in general, see for example [19], [27]. We make a dynamical analysis of 
the linearization of system (M) near each fixed point, which appears to be performant, even for 
the regular solutions. For a better exposition, the proofs are given at Section [TD1 

In Section [5] we study the global existence of G.S. This problem has been often compared with 
the nonexistence of positive solutions of the Dirichlet problem in a ball, see [29], [30] . |12j . [13] . Here 
we use a shooting method adapted to system (M), which allows to avoid questions of regularity of 
system (S). We give a new way of comparison, and improve the former results: 

Theorem 1.1 (i) Assume s < N ^ p ^^p +pa and m < N ^ q ^^ +9b ■ If system (S) has no G.S., then 

(i) there exist regular radial solutions such that X(T) = ^zE and Y(T) = — £^ for some T > 0, 
with < X < and0<Y <^Et on (-oo, T). 

(ii) there exists a positive radial solution (u,v) of the Dirichlet problem in a ball B(0,R). 



This result is a key tool in the next Sections for proving the existence of a G.S. It gives also 
new existence results for the Dirichlet problem, see Corollary 15.31 We also give a complementary 
result: 

Proposition 1.2 Assume s ^ N(p-i)+p+pa m ^ N(q-i)+q+qb ^ y/ien all the regular radial 
solutions are G.S. 

In Section [6] we study the radial solutions of the well known Hamiltonian system 



(SH) 



-Ait = |x| a t; 5 , 
-Aw = \x\ b u^, 



corresponding to p = q = 2 < N, s = m = 0, a > —2, which is variational. In the case a = b = 0, a 
main conjecture was made in 
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Conjecture 1.3 System (SH) with a = b = admits no (radial or nonradial) G.S. if and only if 
(5, jx) is under the hyperbola of equation 



N 



+ 



N 



5 + 1 n+1 



N — 2. 



The question is still open; it was solved in the radial case in [26], [29], then partially in [31] . [9], 
and up to the dimension N = 4 in [33], see references therein. Here we find again and extend to 
the case a, b ^ some results of [20] relative to the G.S., with a shorter proof. We also give an 
existence result for the Dirichlet problem improving a result of 



Theorem 1.4 Let Hq be the critical hyperbola in the plane (5,/j.) defined by 

N — 2. 



N + a + N + b 



5 + 1 n+1 



J1.6) 



Then 



(i) System (SH) admits a (unique) radial G.S. if and only if (6,fx) is above Hq or on "Ho. 

(ii) The radial Dirichlet problem in a ball has a solution if and only if (6,[a) is under Hq. 
(Hi) On 7~Lq the G.S. has the following behaviour at oo : assuming for example 5 > T^rrf, then 



lim r _ 



„N-2 



u(r) = a > 0, and 

lim r (N-2h-(2+b) v = p >0 if ii < 

'I — >oo 

lim r N ~ 2 v = /? > if fi > 
= /3>0 iffi = 



lim r N 2 I In r I 



N + b 

iV-2' 
N + b 
iV-2' 
N + b 

N-2' 



Our proofs use a Pohozaev type function; in terms of the new variables X, Y, Z, W, it contains 
a quadratic factor 



£ H (r) 



II, b 

uv + r ■ 



+ r u 



\ 5+1 N + avu' N + buv' 



+ 



+ 



N-2 

r uv 



XY 



fi + 1 ' ' 5 + 1 5 + 1 r ' fi + 1 r 
Y(N + b-W) (N + a-Z)X~ 



fi + l 



5 + 1 



(1.7) 



As observed in ([20J) the G.S. can present a non-symmetric behaviour. This non-symmetry phe- 
nomena has to be taken in account for solving conjecture (II. 3p . 

In Section [7] we consider the radial solutions of a nonvariational system: 



(SN) 



-Au 
-Av - 



\x\ a u s v s , 

I I A U, S 



where p = q = 2<N,a = b> —2 and m = s > 0. For small s it appears as a perturbation of 
system (SH). In the litterature very few results are known for such nonvariational systems. Our 
main result in this Section is a new result of existence of G.S. valid for any s: 
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Theorem 1.5 Consider the system (SN), with N > 2, a > —2. We define a curve C s in the plane 
(5,/i) by 



N + a N + a Ar n (N - 2)s . . 1 1 
+ ; = N -2 + - ^-!— mm(- 



H + 1 5 + 1 2 > + l'5 + r' 

located under the hyperbola defined by il.6\) . If (5,fj,) is above C s , system (SN) admits a G.S. 



(I- 



This result is obtained by constructing a new type of energy function which contains two terms in 
X 2 ,Y 2 : 



3>(r) = r 



N 



u'v' + r b - 



i^ +l v s n u s v &+l N + avu' N + buv' 



_l_ r 



+ 



+ 



+ 



s vu 



/2 



+ 



s uv 



12 



H+l 5 + 1 5 + 1 r n+1 r 2(5 + 1) it 2(p + l) v 



N-2 

r uv 



XY 



Y(N + b-W) (N + a-Z)X 



H + l 



+ 



X 2 + 



5 + 1 2(5 + 1) 2(// + l) 



-Y 2 



(1.9) 



In Section [8] we consider the radial solutions of the potential system 

-Am = \x\ a u s v m+1 . 



(SP) 



-A q v = \x\ a u s+1 v m , 



where 5 = m + l,fi = s + l and a = b, which is variational, see [34], [35]. Using system (M) we 
deduce new results of existence: 

Theorem 1.6 Let T> be the critical line in the plane (m, s) defined by 

,JV - q , . N — p 

N + a = (m+ 1) + (s + 1) -. 

q p 

Then 

(i) System (SP) admits a radial G.S. if and only if (m,s) is above or on T>. 

(ii) On T> the G.S. has the following behaviour: suppose for example q ^ p. Let X* = N + a — 

pr at N—p 

( s + — m ~q^~\' Then limr-j.oo r p- 1 u(r) = a > 0, and 

N-q 

lim r 9-1 v (r) = /3 > if A* < 0, (1.10) 



lim r %-i-m v ( r ) = p > i/A*>0, (1.11) 



r— >-oo 

V-8 



lim r 8-i |lnr| v(r) = ft > if X* = 0. (1.12) 

In particular il.lO\) holds if p = q, or q 5s m + 1. 

(mj T/ie radial Dirichlet problem in a ball has a solution if and only if (m, s) is under T>. 
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In that case we use the following energy function, which deserves to be compared with the one 
of Section [6] , since it has also a quadratic factor: 



£p(r) = r 



N 



I l\V at I / IP— 2 / I i\q at i / it/ — 2 / 

(s + 1)(^- + ! ) + (m + — J— + ! ) + r a u s+1 v m+l 

p p r q q r 



rjV _ 2 _ >'rVr 1 r zw (« + imjv - P - ^ - i)x) ( ,„ + 1 iz ( a- - ,,-[ q -\ ) v ; 



11 V 



p q 

(1.13) 



Finally in Section [9] we deduce a nonradial result for the potential system in the case of two 
Laplacians: 

-Au = \x\ a u s v m+1 , 
-Av = \x\ a u s+1 v m . 



(SL) 



Our result proves a conjecture proposed in [7j, showing that in the subcritical case there exists no 
G.S.: 

Theorem 1.7 Assume a > —2 and s,m ^ 0. /jf 

s + m + l<nun(— -, - _ - ), (1.14) 

then system (SL) admits no (radial or nonradial) G.S. 

Our proof uses the estimates of [7J, which up to now are the only extensions of the results of 
|18| to systems. It is based on the construction of a nonradial Pohozaev function extending the 
radial one given at (|1.13p for p = q = 2, different from the energy function used in [7J. 

The case of the system (G) with absorption terms (ei = £2 = —1) or mixed terms (ei = —£2 = 
1), studied in [4], [5], will be the subject of a second article. Our approach also extends to a system 
with gradient terms, 



-A p u = ei \x\ a u s v 5 \Vu\ v \Vvf , 
-A q v = e 2 \x\ b u^v m \Vu\ v \Vv\ K , 



"" ' '" " ' n " n ' 1 ' (1.15) 



which will be studied in another work. 

Acknoledgment The authors are grateful to Raul Manasevich whose stimulating discussions 
encouraged us to study system (G). 

2 Reduction to a quadratic system 
2.1 The change of unknowns 

Here we consider the radial positive solutions r 1— > (u(r),v(r)) of system (G) on any interval 
(i?i , i?2 ) ? that means 

(V|f-Vy + ^1 \u'\ p - 2 u' = r^ N (r^ 1 luTV)' = -eir a nV, 
(\v'\ q - 2 A'+Hfl \v'\ q - 2 v' = r 1 ^ (V^ 1 \v'\ p ' 2 i/Y = -e 2 r b u»v m . 
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Near any point r where u(r) ^ 0,u'(r) / and v(r) / 0, v'(r) / Owe define 



X(t) = - — , Y(t) = - — , Z(t) = -e ir 1+a u s v s lu'TV, W(t) = -E 2 r l+b u»v m \v'\~ q v' \ 



TV 



U V 

where t = In r. Then we find the system 

X t = X 
(M)< Y t = Y 



(2.1) 



p— l p— l 
Y - ^2 + W 



q-1 ' q-1 

Z t = Z[N + a-sX-5Y - Z] , 
k W t = W [N + b - fiX - mY - W] . 

This sytem is quadratic, and moreover a very simple one, of Kolmogorov type: it admits four 
invariant hyperplanes: X = 0, Y = 0, Z = 0, W = 0. As a first consequence all the fixed points 
of the system are explicite. The trajectories located on these hyperplanes do not correspond to a 
solution of system (G); they will be called nonadmissible . 

We suppose that the discriminant of the system 



D = Sfj, - (p - 1 - s)(q - 1 - m) ^ 0. 
Then one can express u, v in terms of the new variables: 



(2.2) 



u- 



=r~ 7 ( |Z|) (9 ~ 1_m)/D ( \Y\ q ~ x \W\) S/D , 



(2.3) 



where 7 and £ are defined by 

(p + a)(q - 1 - m) + (q + b)8 
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or equivalently by 



(q + 6)(p - 1 — s) + (p + a)/i 



(p — 1 — 5)7 + p + a = 5£, (q - 1 - m)£ + g + b = ^7. 



(2.4) 



(2.5) 



Since system (M) is autonomous, each admissible trajectory T in the phase space corresponds to 
a solution (u, t> ) of system (G) unique up to a scaling: if (u, v) is a solution, then for any 6 > 0, 
r i-> (9' y u(8r),6^v(6r)) is also a solution. 

2.2 Fixed points of system (M) 

System (M) has at most 16 fixed points. The main fixed point is 



M = (X , Y , Z , W ) = ( 7 ,Z,N-p-(p- 1)7, N — q — (q — 1)£) , 
corresponding to the particular solutions 

u (r) = Ar~ J , v (r) = Br~^, A, B > 0, 



(2.6) 
(2.7) 
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when they exist, depending on e\, e<i- The values of A and B are given by 

A D = {s l7 P- 1 (N -v-lip- I)))"" 1 "" {S2^ q -\N -q-(q- 1)0)' , 
B D = {e 2 ?-\N -q-(q- 1)?))^ {ei^N -p - (p - 1) 7 )) M . 

The other fixed points are 

N - n N — a 

= (0,0,0,0), N = (0,0,N + a,N + b), A = ( f, £,0,0), 

p- 1 q-1 

N — v N — a 

J = ( f, 0,0,0), J = (0, £,0,0), K = (0,0,iV + o,0), L = (0, 0, 0, N + b), 

p — 1 g — 1 

N-v N — v N — a N — a 

Go = ( f , 0, 0, N + b f M ), F = (0, f TV + a £<5, 0), 

p—l p — 1 <? — 1 

and if m 7^ 5 — 1 , 

D J\T-p £fM-(ff + «0 n (9-l)(iV + 6-fEf/x)-m(iV-g) 

m) = ( r> ; ' U ' 



p — 1 q — 1 — m q — 1 — m 

(i q + 6 )0) (iV + 6)(g-l)-m(JV 



— 1 — m q — 1 — m 



g — 1 — m' g — 1 — m' q — 1 — m 

and by symmetry, if s 7^ p — 1, 

„ ,£?«-(P + a) iV-q (p-l)(iV + a-^)- S (iV-p) 

^0 = ( : , : ,0), 

p — 1 — s q — 1 p — 1 — s 

D „ = C_ " + ° |0t (Af + -)(p-l)-(^-rt i0 ^ , 
\ p — 1 — s p — 1 — s / 

5b =f- p : a ,o, (iv+a)(p -! ) - g(Ar - y) ,iv + 6 + ^- a+p 



p — 1 — s p — 1 — s p — 1 — s 

2.3 First comments 

Remark 2.1 T/iis formulation allows to treat more general systems with signed solutions by re- 
ducing the study on intervals where u and v are nonzero. Consider for example the problem 

A 1 1 a 1 is 1 i(5—l a 1 \b 1 im 1 lit— 1 

pli = £1 |x| |«| |f I V, —AqV = 62 \X\ \v\ \Uf U. 

On any interval where uv > 0, the couple (\u\,\v\) is a solution of (G). On any interval where 
u > > v, the couple (u, \v\) satisfies (G) with (£1,62) replaced by (— £1, — £2)- 

Remark 2.2 There is another way for reducing the system to an autonomous form: setting 
U(t) = r 7 n, V(t) = r*«, H(t) = -r^ 1 )^" 1 ) \u'\ p ~ 2 u' , K(t) = -r^X'" 1 ) IwT'V, 
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with t = In r, we find 



f U t = 7 U - IHI^)/^ 1 ) H, V t = CU - | K |(2-^)/C Q -i) K , . , 

1 H t = ( 7 (p - 1) + p - N)R + £l U s V 5 , K t = (C(q -l)+q- N)K + e 2 WY m . ( ' ' 

It extends the well-known transformation of Emden- Fowler in the scalar case when p = 2, used also 
in for general p, see Section [5] When p = q = 2 we obtain 

U tt + (N - 2 - 2 7 )U t - 7 (iV - 2 - 7 )U + eilPV 5 = 0, 
V tt + (JV - 2 - 2£)V t - £(JV - 2 - £)V + e 2 lPV m = 0, 1 " ' 

which was extended to the nonradial case and used for Hamiltonian systems (s = m = 0), with 
source terms in J2C (e\ = £2 = 1) and absorption terms in [J^l (e% = e 2 = — 1)- Our system is more 
adequated for finding the possible behaviours: unlike system \2. 8\) it has no singularity, since it is 
polynomial, also its fixed points at 00 are not concerned when we deal with solutions u, v > 0. 

Remark 2.3 In the specific case p = q = 2, setting 

2 + a L,l s ~ 2 „, lo,! 5-1 „, „„ — VVtf — <~ „ 2 +6 1 , |M— 1 „ \ n ,\m— 2 



v ry 2+a i is— 2 i |(5—1 Trjj; 2+6 i i it — 1 

z = aZ = e\r \u\ u \v\ v, w = YW = e 2 r kt 



U V V, 



we get the following system 

X t = X 2 — (N — 2)X + z, Y t = Y 2 -(N- 2)Y + w, 
zt = z [2 + a + (1 - s)X - 5Y] , w t = w [2 + b - pX + (1 - m)Y] . 

It has been used in 120}/ for studying the Hamiltonian system (SH). Even in that case we will show 
at Section^ that system (M) is more performant, because it is of Kolmogorov type. 

Remark 2.4 Assume p = q and a = b. Setting t = kt and (^X, Y, Z, Wj = k(X,Y, Z,W), we 
obtain a system of the same type with N, a replaced by N, a, with 

N-p_ k _N+a 
N — p ~ ~ N + a' 

It corresponds to the change of unknowns 

r = r k , u(f) = C lU (r), v(r) = C 2 u(r), Cl = k P(p-i-m+S)/D ^ ^ = k ( P ( P -i-s+^)/D ^ 

From \2. 3\) and \2.1$ , we get 7/7 = £/£ = k = |±£. There is one free parameter. In particular 
1) we get a system without power (a = 0), by taking 

N = * N + a \ k P ■ 



p + a p + a 

2) we get a system in dimension N = 1, by taking 

p-1 „ _ p + a- (N + a)p 

Tt ^ ' ' a ~ AT ' 

Jy — p Jy — p 
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3 The scalar case 



We first study the signed solutions of two scalar equations with source or absorption: 

A 1-N ( N—l I / IP— 2 A' | , a | iQ-l fn -i\ 

— A p u = — r Ir \u \ uj =e\x\ |up u, (3.1) 

with e = ±1, 1 < p < iV, Q j^p—1 and p + a > 0. 

We cannot quote all the huge litterature concerning its solutions, supersolutions or subsolutions, 
from the first studies of Emden and Fowler for p = 2, recalled in [16J; see for example [2] and [37J, 
for any p > 1, and references therein. We set 

(N + a)(p-l) N{p-l)+p + pa p + a 

c » 1 ~~ at ' Q2 — T7 , 7 — 7m • 

N — p N — p Q + 1 — p 

From Remark l2.4l we could reduce the system to the case a = 0, in dimension iV = p(N + a)/(p+a). 
However we do not make the reduction, because we are motivated by the study of system (G), and 
also by the nonradial case. 



3.1 A common phase plane for the two equations 

Near any point r where u(r) ^ (positive or negative), and u'(r) ^ setting 



X (t) = -—, Z(t) = -er 1+a \u\ Q - l u\u'\~ p u', (3.2) 
u 1 1 

with t = lnr, we get a 2-dimensional system 

(M scal) 




Y - E=2 J- -Z- 

^ p-l "f" p-1 



: X 

Z[N + a-QX - Z 



and then \u\ =r~~ t ( \Z\ \X\ p ^ 1 ) 1 /^ +1 ~ p \ This change of unknown was mentioned in [TT] in the case 
p = 2, s = 1 and N = 3. It is remarkable that system (M sca i) is the same for the two cases e = ±1, 
the only difference is that X(t)Z(t) has the sign of e : 

The equation with source (e = 1) is associated to the I s * and 3 rd quadrant. It is well known that 
any local solution has a unique extension on (0, oo) . The I s * quadrant corresponds to the intervals 
where \u\ is decreasing, which can be of the following types (0, oo) , (0, i?2),(-Ri, oo),(i?i, R2), < 
R\ < i?2 < 00. The 3 rd quadrant corresponds to the intervals {R\,R,2) where \u\ is increasing. 

The equation with absorption (e = —1) is associated to the 2 nd and A th quadrant. It is 
known that the solutions have at most one zero, and their maximal interval of existence can be 
(0,i?2), (Ri,oo), (i?i,i?2) or (0,oo). The 2 nd quadrant corresponds to the intervals (R±,R2) where 
\u\ is increasing. The 4 th quadrant corresponds to the intervals (0, R2) or (i?i,oo) where \u\ is 
decreasing. 

The fixed points of (M sca i) are 

N — v 

M = (X ,,Zo) = ( 1 ,N-p-(p-l)j), (0,0), N = (0,N + a), A = ( f,0). 

p — 1 
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In particular Mq is in the I s * quadrant whenever 7 < ■— f , equivalently Q > Qi, and in the 4' 
quadrant whenever Q < Q\. It corresponds to the solution 

u(r) = Ar~^, for e = 1, Q > Q 1: or e = -1, Q < Qi, 
where A = (e 7 p - 1 (^ r ~P ~ 7(p " 1))) 1/(Q ~ P+1) . 



3.2 Local study 

We examine the fixed points, where for simplicity we suppose Q ^ Qi, and we deduce local results 
for the two equations: 

• Point (0, 0) : it is a saddle point, and the only trajectories that converge to (0, 0) are the 
separatrix, contained in the lines X = 0, Y = 0, they are not admissible. 

• Point Nq : it is a saddle point: the eigenvalues of the linearized system are and — N. the 
trajectories ending at Nq at 00 are located on the set Z = 0, then there exists a unique trajectory 
starting from —00 at Nq; it corresponds to the local existence and uniqueness of regular solutions, 
which we obtain easily. 

• Point Aq : the eigenvalues of the linearized system are and ^E\(Qi ~ Q)- If Q < Qi, 
Aq is an unstable node. There is an infinity of trajectories starting from Aq at —00; then X(t) 

N— N ~ p 

converges exponentially to ■— ^ , thus lim r _j.o r u = a > 0. The corresponding solutions u satisfy 
the equation with a Dirac mass at 0. There exists no solution converging to Aq at 00. If Q > Qi, Aq 
is a saddle point; the trajectories starting from Aq at — 00 are not admissible; there is a trajectory 

JV-p 

converging at 00, and then lim^oo r p- 1 u = a > 0. 

• Point Mq : the eigenvalues Ai, A2 of the linearized system are the roots of equation 

A 2 + (Z - X )X + Q ~ P ^ 1 X Z = 0. 
p — 1 

For e = 1, Mq is defined for Q > Q\\ the eigenvalues are imaginary when Xq = Zq, equivalently 
7 = (N — p)/p, Q = Q2- When Q < Q2, Mq is a source, there exists an infinity of trajectories 
such that lim r _s.o r 7 n = A. When Q > Q2, Mq is a sink, and there exists an infinity of trajectories 
such that lim r ^.qo r 7 n = A. When Q = Q2, Mq is a center, from [2] For e = — 1, Mq is defined 
for Q < Qi, it is a saddle-point. There exist two trajectories 7~i,7~{ converging at 00, such that 
Hindoo r 7 u = A and two trajectories 7i, 7^', converging at 0, such that lim r _ > .o r 7 n = A. 



3.3 Global study 

Remark 3.1 System (M sca i) has no limit cycle for Q ^ Q2. It is evident when e = — 1. When 
e = 1, as noticed in }19$ . it comes from the Dulac's theorem: setting Xt = f(X, Z), Zt = g(X, Z), 
and 

B(X, Z) = x pQ ^ Q+1 ~^- 2 Z^^ Q+1 -^- 1 , M = B x X t + B z Z t + B(f x + g z ), 
then M = KB with K = (Q2 — Q)"f(N — p)/p, thus M has no zero for Q ^ Q2- 
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Then from the Poincare-Bendixson theorem, any trajectory bounded near ±00 converges to one 
of the fixed points. Thus we find again global results: 

• Equation with source (e = 1). If Q < Q\, there is no G.S.: the regular trajectory T issued 
from iVo cannot converge to a fixed point. Then X tends to 00 and the regular solutions u are 
changing sign, there is no G.S.. 

If Qi < Q < Q2, the regular trajectory T cannot converge to Mq; if it converges to Aq, it is 
the unique trajectory converging to Aq; the set delimitated by T and X = 0, Z = is invariant, 
thus it contains Mq; and the trajectories issued from Mq cannot converge to a fixed point, which 
is contradictory, then again X tends to 00 on T and the regular solutions u are changing sign.. 
The trajectory ending at Aq converges to Mq at —00; then there exist solutions u > such that 

N-p 

lim r _>o r ^ u = A and lim r ^o r u = a > 0. 

If Q > Q2, the only singular solution at is uq, and the regular solutions are G.S., with 
limj._5.0o r 7 -u = A. Indeed Mq is a sink; the trajectory ending at Aq cannot converge to iVo at —00, 
thus X converges to 0, and Z converges to 00, then u cannot be positive on (0, 00). The trajectory 
issued from Nq converges to Mq. 

• Equation with absorption (e = — 1). If Q > Qi, all the solutions u defined near are regular; 
indeed the trajectories cannot converge to a fixed point. 

If Q < Qi, we find again easily a well known result: there exists a positive solution u\, unique 

N-p 

up to a scaling, such that lim T -_ ) .o r p- 1 u\ = a > 0, and lim^oo r 7 ui = A. Indeed the eigenvalues 
at Mq satisfy Ai < < A2. There are two trajectories Ti,T{ associated to Ai, and the eigenvector 
(Xq + I Ai| , — The trajectory T\ satisfies X t > > Z t near 00, and X > -^-f , since Zq < 0, 

and X cannot take the value because at such a point Xt < 0; then < X < Xq and 
Xt > as long as it is defined; similarly Zq < Z < and Z t < 0; then 7i converge to a fixed point, 
necessarily Aq, showing the existence of u\. The trajectory 7~{ corresponds to solutions u such that 
linij.-j.oo r 7 -u = A and linv-^ii = 00 for some R > 0. There are two trajectories 7i, 7^, associated 
to A2, defining solutions u such that lim r _ > o r7u = A and changing sign, or with a minimum point 
and lim r _>/j u = 00 for some R > 0. The regular trajectory starts from iVo in the 2 nd quadrant, it 
cannot converge to a fixed point, then lmv-^u = 00 for some R > 0. 

• Critical case Q = Qi '■ it is remarkable that system (M sca i) admits another invariant line , 
namely AqNq, given by 

X Z N-p , , 

- + 77— r = 0. (3.3) 

p Q2 + 1 P 

It precisely corresponds to well-known solutions of the two equations 
U = C(K 2 + r (P+»)/(p-l))(p-iV)/(p+a) )for £=1 . u = cK 2_ r (p+a)/(p-l) for £ = _^ 

where K 2 = c Q - p+1 {N + a)^ 1 {(N - p)/(p - l)) 1_p . 

Remark 3.2 The global results have been obtained without using energy functions. The study of 
J1J/ was based on a reduction of type of Remark \2.%h using an energy function linked to the new 
unknown. Other energy functions are well-known, of Pohozaev type: 



T a (r) = r 



N 



\u \ „ \u\ u\u\ u 
+ i 7T — t + o-— — 



p' Q + l 



r N ~ p \u\ p \X\ p - 2 X 



X Z 

7 + OTT ~ c r 
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with a = satisfying F' a {r) = r N ~ l+a ( g±f - ^2 J |u| Q+1 , or with a = g±f , /eadin 5 to 

F' a (r) = r N ~~ x {j^f^t — ^ e critical case Q = Q2, all these functions coincide and 

they are constant, in other words system (M sca i) has a first integral. We find again the line $3.3\) : 
the G.S. are the functions of energy 0. 

4 Local study of system (S) 

In all the sequel we study the system with source terms: (G) = (S). Assumption (jl.5p is the most 
interesting case for studying the existence of the G.S. 

We first study the local behaviour of nonnegative solutions (u, v) defined near or near 00. It 
is well known that any solution (it, v) positive on some interval (0, R) satisfies u' , v' < on (0, R) . 
Any solution (u,v) positive on (R,oo), satisfies u',v' < near 00. We are reduced to study the 
system in the region 1Z where X, Y,Z,W > 0, and consider the fixed points in 1Z. Then 

ru l rv > r l+a s 8 r l+b m fi 

= y(t) = "V ZW = ^F^' w{t) = 1 ; (41) 

U V \u'[ \v \ H 

and (X, Y, Z, W) is a solution of system (M) in 1Z if and only if (u, v) defined by 

u=r-' t (ZX p - 1 ) < - q - 1 - m ^ D (WY q - 1 )^ D , v^-^WY^^P-^^iZXP- 1 )^ (4.2) 

is a positive solution with u',v' < 0. Among the fixed points, the point Mq defined at (|2.6p lies in 
7Z if and only if 

N-p N -q , . 

< 7 < f and < £ < ^. (4.3) 

p—1 q — 1 

The local study of the system near Mq appears to be tricky, see Remark 14.21 A main difference 
with the scalar case is that there always exist a trajectory converging to Mq at ±00 : 

Proposition 4.1 (Point Mq) Assume that |^.3[ ) holds. Then there exist trajectories converging to 
Mq as r — > 00, and then solutions (u,v) being defined near 00, such that 

lim r 7 n = a > 0, lim r^v = /3 > 0. (4-4) 

There exist trajectories converging to Mq as r — > 0, and thus solutions (u, v) being defined near 
such that 

lim r 7 n = a > 0, lim r^v = (3 > 0. (4-5) 

1 — >0 r— s-0 

Proof. Here Mq € K; setting X = X + X, Y = Y + Y, Z = Z + Z, W = W + W, the 
linearized system is 

x t = x (x + ¥ ^z), 

Y t = Y (Y + ^ I W), 
Z t = Zq(-sX -5Y- Z), 
W t = Wo(-fiX -mY- W). 
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The eigenvalues are the roots Ai, A2, A3, A4, of equation 



(\-X )(\ + Z ) + ^-X Z 
p — 1 

(4-6) 

This equation is of the form 

/(A) = A 4 + EX 3 + FX 2 + GX- H = 0, 

with 

E = Z - X + W - Y , 
F = (Z - X )(Wo - Y ) - ^X Z - m^k Yo W , 
< G= _2^ YoWo{Zo _ Xo) _E_k T l Xo z (W -Y ), 

H = j^^XqYqZqWq. 

From fjl .5[) we have H > 0, then A1A2A3A4 < 0. There exist two real roots A3 < < A4, and 
two roots Ai,A2, real with A1A2 > 0, or complex. Therefore there exists at least one trajectory 
converging to Mq at 00 and another one at —00. Then (|4.4p and (|4.5h follow from (14. 2D . Moreover 
the convergence is monotone for X, Y, Z, W. ■ 



rn 



(A-y )(A + ^ ) + rYoWt 



Sfj, 

(p-l)(g-l 



-X Y Z W = 0. 



Remark 4.2 There exist imaginary roots, namely ReAi = ReA2 = 0, if and only if there exists 
c > such that f(ci) = 0, that means Ec 2 — G = 0, and c 4 — Fc 2 — H = 0, equivalently 

E = G = 0, or EG > and G 2 - EFG - E 2 H = 0. 

Condition E = G = means that 

(i) either Zq = Xq and Wo = Yq, i.e. 

N — v N — a 

(7,0 = (^A^), (4.7) 

*n other words (o» = (^^f^, p^-i-^ji-n^)) } 

(ii) or (p — 1 — s)(q — 1 — m) > and (7, £) satisfies 

2N - p - q = jry + g£, 

(1 - ^r)e(iv -(?-(<?- 1)0 = (1 - ^t)7(^ - p - (p - 1)7). 



(4.8) 



JTiis gives m general 0,1 or 2 values of (7, £)■ ^ or example, in the case = — 7^ 1, and 
(p — 2)(g — 2) > and N > ^Sp2 we fi n d another value, different from the one of (f^.Tp /or p ^ q : 

o-2 p - 2 

( 7 ,£ =(J\T— ^ 1,7V— ^ 1). (4.9) 

pq — p — q pq — p — q 

Moreover the computation shows that it can exist imaginary roots with E, G 7^ 0. 



In the case p = q = 2 and s = m the situation is interesting: 
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Proposition 4.3 Assume p = q = 2 and s = m < jfz^, with 5 + 1 — s > 0, /x + 1 — s > 0. In the 

plane (5,fi), let 7i s be the hyperbola of equation 

1 1 N-2 



5 + l-s n + l-s N-(N-2)s' 

equivalently 7 + £ = N — 2. Then % s is contained in the set of points for which the linearized 
system at Mq has imaginary roots, and equal when s f$ 1. 

Proof. The assumption D > imply 5 + 1 — s > and fi + 1 — s > 0; condition £7 = G = 
implies s < N/(N — 2) and reduces to condition (|4.10p . Moreover if s 1, all the cases are covered. 
Indeed 2G = (s - 1)£ [Yb^o + *qW ] , hence G-E ^ 0. ■ 

Next we give a summary of the local existence results obtained by linearization around the 
other fixed points of system (M) proved in Section [TU1 Recall that t — > —00 as r — > and t — > 00 
as r — > 00. 

Proposition 4.4 (Point Nq) A solution (u,v) is regular if and only if the corresponding trajectory 
converges to Nq when r — > 0. For any uq,vq > 0, there exists a unique local regular solution (u,v) 
with initial data (uo,vo). 

Proposition 4.5 (Point A ) If s^f + 5^ > N + a and fj,^ + m^r > N + 6 > there exist 
(admissible) trajectories converging to Aq when r — > 00. If s-^rf + $i^f < N + a and H^Ei + 
m E^l <N + b, the same happens when r — > 0. In any case 

N — p N — q 

limr p- 1 u = a: > 0, limr 1- 1 v = (3 > 0. (4-11) 



// s— ^ + < N + a or + m lf^i < N + b, there exists no trajectory converging when 

N- 

P- 



r — > 00; i/ + > -/V + a or /i^rf + m ^~i > N + b, there exists no trajectory converging 



when r — > 0. 



Proposition 4.6 (Point Pq) 1) Assume that q > m + 1 and g + 6 < — — f/x < iV + 6 — m^^. If 

7 < taere exist trajectories converging to Pq when r — > 00 ('and not w/ien r — )■ 0). If"f> 
the same happens when r — > (and not w/ien r — > 00). 

Assume that q < m + 1 and q + b > ^ff/i > iV + 6 - m^f^ and q^Ej^ + m{N - q) ^ 
N(q — 1) + (0 + 7/ 7 < t/iere exist trajectories converging to Pq when r — > C and not waen 
r — > 00). 1/7 > t/iere exist trajectories converging when r — > 00 ('and not u>aen ifften r — >■ 0). 

In any case, setting k = g _ 1 1 _, m ( ^E± I 1 ~ (,1 + ^))? ^ fter e ao/ds 

jV-p 

limr p- 1 n = a > 0, limr K u = /3>0. (4-12) 

Remark 4.7 Tftis result improves the results of existence obtained by the fixed point theorem in 
[27^ in the case of system (RP) with p = q = 2, a = 0, N = 3, 2s + m ^ 3. The proof is quite 
simpler. . 
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Proposition 4.8 (Point Iq) If -^zrfs > N+a and -^z^/i > N+b, there exist trajectories converging 
to Iq when r — > oo, and t/ien 

N-p 

lim r p- 1 u = (3, lim v = a > 0. (4-13) 

r— >oo r—^oo 

For any s, m ^ 0, there is no trajectory converging when r — > 0. 

Proposition 4.9 fPoini G ) Suppose j=^H < N + b. If q + b < /i and N + a < s, t/iere 

exist trajectories converging to Go when r — > oo. // ^j-^ < 9 + ^ an d ^rr s < iV + a, tae same 
happens when r — > 0. in any case 

Af-p 

lim rp- 1 u = {3, limv = a > 0. (4-14) 



Proposition 4.10 (Point Co) Suppose N + b < (hence q < m + 1) u>ita m 7^ ^ 9 7v~g b+1 ^ 9 > 

and 5 > ( jV + a K^"+ 1 g) _ 77^^ i/iere exist trajectories converging to Co when r — > 00 (and not when 
r — > 0), and taen 

limu = a>0, limr fc ?; = /3, (4.15) 



where k 



q+b 
m+l — q ' 



Proposition 4.11 (Point Ro) Assume that N + b < ^f^ m (hence q < m + 1) wit/i m 7^ 
N< ^ q ~N-q b+bq > anc ^ ^ < (N+aX-m+i-g) _ (p+n)(m+i g) ^ ^ i/jere exist trajectories converging to 
Ro when r — > 00 ('and not w/ien r — > 0). If 5 < (p+ a )( ? ^+ 1 g) ; f^e^ exist trajectories converging 
when r — > ('and not waen r — > 00), and t/ien i5[ ) ao/ds again. 

We obtain similar results of convergence to the points Qo, Jo, Ho, Do, So by exchanging p, 5, s, a 
and q, fi, m, b. There is no admissible trajectory converginf to 0, Kq, Lq, see Remark llU.il 

5 Global results for system (S) 

We are concerned by the existence of global positive solutions. First we find again easily some 
known results by using our dynamical approach. 

Proposition 5.1 Assume that system (S) admits a positive solution (u,v) in (0, 00). Then the 
corresponding solution (X, Y, Z, W) of system (M) stays in the box 

A = (0, EzJL \ x (0, x(0,N + a)x (0, N + b) , (5.1) 



p-l 



in other words 



ru > + ?L_P u>0i rV ' + — — %>0, r l+a u s v 5 <{N + a)\u'\ p ~ 1 , r 1+b u»v m < (N + b)\v'\ q ~ 1 . 
p-l q-1 

(5.2) 
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and then 

u s-p+i v s ^ c ir -(p+«) j u i*v m - q+1 S C 2 r- [q+b \ in (0,oo), (5.3) 
^ere d = (JV + a)(f^)P-\ C 2 = (iV + &)(£=£)«-i, and 

N — p N — q N — p N — q 

lim r p- 1 u = c\ ^ 0, lim r i- 1 v(r) = c 2 ^ 0, lim inf r p- 1 u > 0, lira inf r i- 1 v > 0. (5.4) 

r— >0 r— s-0 ~ r— >oo r— voo 

As a consequence if s ^ p — 1 or m ^ q — 1, we have 

u 5= ATir~ 7 , v ^ °% m(0, oo), (5-5) 
witt ifi = dT x -^ ID (?J D ,K* = C» ID ct l ~ s)/D - 

Proof. The solution of system (M) in 7?. denned on R. On the hyperplane X = — 5^ we have 
X< > 0, the field is going out. If at some time to, X(to) = , then A"(i) > for t > to, 

> 0, since since Z > 0, thus X(t) > 2-^p for t > t\ > to, then 
^ X 2 /2, which implies that X blows up in finite time; thus X{t) < on in the same way 
Y(i) < ^rj*. On the hyperplane Z = JV + a we have < 0, the field is entering. If at some time 
t , Z(to) = N + a then Z(t) > N + a for t < t , then Z f ^ Z(AT + a - Z), since sA + 51" > 0, and 
Z blows up in finite time as above; thus Z(t) < N + a on M, in the same way W(t) < N + b. Then 
(|5.2p . (|5.3p and (|5.5p follows. By integration it implies that A N ~ p ^^ 1 ^u(r) is nondecreasing near 
or oo, hence (|5.4p holds. ■ 

Next we prove Theorem I l.li 

Proof of Theorem ll.lL (i) The trajectories of the regular solutions start from iVo = (0, 0, iV + 
a, N + b), from Proposition 14.41 and the unstable variety V u has dimension 2, from (jlO.ip . (|10.2p . 
It is given locally by Z = <p(X, Y), W = ip(X, Y) for (X, Y) G B(0, p)\ {0} C E 2 . 

To any (x,y) £ B(0, p)\{0} we associate the unique trajectory T x , y in V u going through this 
point. If T* is the maximal interval of existence of a solution on T x , y , then limj_ > r* (X(t) + Y(t)) = 
oo. Indeed Z, and W satisfy 0<Z<N + a, 0<W<N + bas long as the solution exists, 
because at a time T where Z(T) = N + a, we have Z% < 0. If there exists a first time T such that 
X(T) = ^£ or Y(T) = 2=2, then T < T*. We consider the open rectangle A/" of submits 

/ s /N-p \ / N-q\ fN-p N-q\ 

(o,o), „ = (_J!,oj, ^o,— ij, 

Let W = {(x, y) € B(0, p) : x,y > 0}; then U = Si U S 2 U 5 3 U 5, where 

f iSj = {(x,y) G U : %, y leaves A/" on (wi,w)} , i = l,2, 

\ Ss = {(x,y) £ U : T x , y leaves J\T at -cu} , S = {(x, y) G U : T x , y stays in J\f} . 

Any element of S defines a G.S. Assume s < N (P~^)+P+P a m L e t us s how that 5i is nonempty. 
Consider the trajectory T&,o on V u associated to (x, 0), with x G (0,p) , going through M = (x,0, 



in turn X t ^ A 



A 



iV- 
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<p(x, 0),if)(x, 0)); it is not admissible for our problem, since it is in the hyperplane Y = 0: it satisfies 
the system 

X t = x\x -?tz2+z 
L p— l p— l 

Z t = Z [JV + a - sX - Z 

which is noi completely coupled. The two equations in X, Z corresponds to the equation 

- A P U = r a U s . (5.6) 

The regular solutions of (|5.6|) are changing sign, since s is subcritical, see Section [3j Consider the 
solution (X ,Y , Z ,W) of system (M), of trajectory Tx,o, going through M at time 0; it satisfies 
Y = 0, and > 0, Z(i) > tend to oo in finite time T*, then for any given C > — 5p , there 
exist a first time T < T* such that X(T) = C, and F(T) = 0. We have lim t _ > _ o0 TV = N + 6, and 
necessarily < W < N + b, in particular < W(T) < N + b; and Wj is bounded on (— oo,T*) , then 
W" has a finite limit at T*. The field at time T is transverse to the hyperplane X = we have 

Xt = > 0, since -Z(T) > 0. From the continuous dependance of the initial data at time 0, 

for any e > 0, there exists r\ > such that for any (x, y) G B((x, 0), rj) and for any (X, Y, Z, W) on 
T x ,y, there exists a first time T £ such that -X"(T e ) = C, and |Y(t)| ^ e for any t ^ T e , in particular 
for any (x, y) G B((x, 0), ry) with y > 0, and then < Y(i) < e for any t < T e . Let us take C = • 

Then (x, y) £ (Si. The same arguments imply that S\ is open. Similarly assuming m < N ^ q ~^^ !+qb 
implies that 52 is nonempty and open. By connexity <S is empty if and only if £3 is nonempty. 

(ii) Here the difficulty is due to the fact that the zeros of u, v correspond to infinite limits 
for X, Y, and then the argument of continuous dependance is no more available. We can write 
U = Mi U M 2 U M 3 U S, where 

M\ = {(x, y) £li and T X) y has an infinite branch in X with Y bounded} , 
Mi = {(x,y) : T x ,y has an infinite branch in Y with X bounded} , 
M3 = {(x, y) : T x , y has an infinite branch in (X, Y)} . 

In other words, Mi is the set of (x,y) G U such that for any (X,Y, Z,W) on T x , y , there exists 
a T* such that lim^y* X(t)) = 00, and Y(t) stays bounded on (— 00, T*) , that means the set of 
(x, y) G U such that for any solution (u,v) corresponding to T x , y , u vanishes before v; similarly for 
M 2 - Otherwise Ms is the set of (x,y) G U such that there exists a T* such that lim^y* X(t) = 
lim^T* Y{t) = 00, that means (u,v) vanish at the same R* = e T " . In that case, from the Hopf 
Lemma, lim r _> fl ^ r ^ u = 1, then lim t ^T* y = I. 

We are lead to show that Mi is nonempty and open for s < N (p~^)+P+P a _ \y e consider again 

the trajectory T and take C large enough: C = 2(^Ey + N q t}i )■ Let e G (0, ^) . For any (x,y) G 
B((x, 0), 77) with y > 0, and any (X, Y, Z, W) on 7^ l3/ , there is a first time T e such that X(T £ ) = C, 
and < Y(t) ^ e for any t T e . And X is increasing and Xt ^ X(X — C), thus there exists T* 
such that limt-tT* X(t) = 00. Setting ip = X/Y, we find 

ip p — 1 q — 1 g — 1 p — 1 2 



18 



then (pt(T e ) > 0. Let 9 = sup{i > T E : cp t > 0} ; suppose that 9 is finite; then <p(9) > f(T e ) = 
C/e > 2 and X (9) ^ Y (6) + C < X (0) /2 + C, which is contradictory. Then ip is increasing up 
to T*; if lim^T* Y(t) = oo, then lim^T* y = 1, which is impossible. Then (a;,y) € .Mi, thus .Mi 
is nonempty. In the same way A4\ is open. Indeed for any (x,y) € .Mi there exists M > such 
that < Y(i) < M/2 on 7i,g. To conclude we argue as above, with (x, 0) replaced by (x,y), and 
C replaced by C + M. m 

Proof of Proposition [TT2~l Assume s ^ N (p-^)+p+P a _ Consider the Pohozaev type function 



,N 



,l\P 



+ 



P 



We find 7"(0) = and 
F'(r) 



r a u s+1 r N — pu\u'\ p 2 u' 

-v d + £l—LJ 

s+1 p r 



1 + s + 1 



Z 



N -p 
P . 



r 



N-l+a 



r N-l+a v 5 u s+l 



N + a N -p 
s + 1 p 

N + a N-p 



v s u s+1 + 



s + 1 



s+1 6-1 i 
-ru v v 



s + 1 



P 



5Y 
s + 1 



(5.7) 



(5.8) 



From our assumption, T is decreasing, and Z > 0, thus X < ^j 3 . Then S\,Ss are empty. If 
moreover m ^ N ^ q ~^^ +qh then 52 is empty, therefore S = 14. ■ 

Remark 5.2 Lei us on/y assume that s ^ AT(p-i)+p+pa ^ function has a first zero, it is 

v. Indeed if there exists a first value R where u(R) = 0, and v(r) > on [0,R) , then J~(R) = 
f-\u'(R)\ p >0. 

As a first consequence we obtain existence results for the Dirichlet problem. It solves an open 
problem in the case s > p — 1 or m > q — 1, and extends some former results of [12] and [12] . Our 
proof, based on the shooting method differs from the proof of [12], based on degree theory and 
blow-up technique. Our results extend the ones of [3[ Theorem 2.2] relative to the case p = q = 2, 
obtained by studying the equation satisfied by a suitable function of u, v. 



Corollary 5.3 system (S) admits no G.S. and then there is a radial solution of the Dirichlet 
problem in a ball in any of the following cases: 

(i) p<8 + l,q<m + l, and min(s^ + ^6 - (N + a), + - (N + b)) ^ 0; 

(ii) p<s + l,q>m+l and s^ff + ^5 - (N + a) ^ or 7 - £=f > 0; 
(Hi) p > s + 1, q > rn + 1 and max(7 — £ — j_ zi) ^ 0; 



fn?) p ^ s + 1, q ^ m + 1 and max(7 



TV-p p 
v-l ' s 



9- 

q-1 ) 



> 0. 



Proof. From Theorem II .![ we are reduced to prove the nonexistence of G.S. 



(i) Assume p < s + 1, and s 



Af-p 1 N-q 



p-1 q-1 

From [6J, Theorem 3.1], we find u = 0(r ^ p+a 



5 - (N + a) < 0. We have -A p u ^ Cr° 



t- 1 u s for large r. 



JV- 9 
9 



? a)/(s+1 - p) ), and then s^ + ^S-(N + a) ^ 0, 
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from (|5.4p . which contradicts our assumption. In case of equality, we find —A p u ^ Cr~ N for large 
r, which is impossible. Then there exists no G.S. This improves ythe result of [12] where the 
minimum is replaced by a maximum. 

(ii) Assume p < s + 1, q > m + 1 and 7 — > 0; then u = 0(r -7 ), which contradicts (|5.4p . If 

■pr N — p ^ N — p 

7 — = 0, then limr p- 1 u = a > 0, and £ > -^-f • Hence — A g u > Cr p-i ^v m for large r, then 
v > (j r (i +b ~ p-i S)/{q-l-m) _ Q r -(, _ There exists Ci > such that Cj ^ r^v < 2Ci for large r, from 
[61 Theorem 3.1] and (|53]> . then -A p u ^ Cr"^ for some C > 0, which is again contradictory. 

(iii) (iv) The nonexistence of G.S is obtained by extension of the proof of [12] to the case a, b 7^ 0. 
Moreover (iii) implies the nonexistence of positive solution (u,v), radial or not, in any exterior 
domain (R, 00) x (R, 00), R > from [6]. ■ 

Corollary 5.4 Assume [pfy with p = q = 2. If 5 + s ^ ^j^t, 2 " and fjL + m ^ jV ^ 2 + 2fe , tten system 
(5) admits a G.S. 

Proof. It was shown in [28] , [H] by the moving spheres method that the Dirichlet problem has 
no radial or nonradial solution. Then Theorem 11.11 applies again. ■ 

We aso extend and improve a result of nonexistence of [10] for the case p = q = 2, a = 0, s > 1: 
Proposition 5.5 Assume s + 1 > p or 7 > ^77^ > an d 

p(N -q) ^ < iVQ- 1) +pq + p ^ g ^ 



(g-l)(JV-p) iV-p 

T/ien system (S) admits no G.S. and then there is a solution of the Dirichlet problem. The same 
happens by exchanging p, s, 5, a, 7 with q, m, /i, b, £. 

Proof. Consider the function T defined at (|5.7p . Suppose that there exists a G.S. Then from 
(JED and ([52]) we find 

N + a N -p 5Y N + a N -p 5 N — q ^ 

L > tL > 0. 

s + 1 p s + 1 s + 1 p s + 1 q — 1 

From (I5.8p . we deduce that T is nondecreasing. First suppose s + 1 > p. From (15. 3D and (I5.4p .it 
follows that u = 0(r~ fc ) at 00, with k = (p+o-ff^f )/(s-p+l). In turn r^-^V = 0(r^ N - p ^- kp ) = 

o(l) from (|5.9|) . then .F(r) = o(l) near 00. Next assume s + 1 5= p and 7 > —^2. Then r N ~ p u p = 
0(r N ~ p ~ rp ), hence ^(r) = o(l) near 00. In any case we get a contradiction. ■ 

6 The Hamiltonian system 

Here we consider the nonnegative solutions of the variational Hamiltonian problem (SH) in C 



(SB) 



A 1 ia 5 
It = \X\ V , 

-Aw = Ixl it*\ 
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where p = q = 2 < N, s = m = 0, a > b > —2, and D = Sfj, — 1 > 0. For this case we find 

(2 + a) + (2 + b)5 _ 2 + b+(2 + a)n ^ _ 

1=- -, i= r — , 7 + 2 + a = 5£, £ + 2 + b = ^. 



D 



D 



The particular solution (uo(r), vo(r)) = (Ar 7 ,Br exists for < 7 < iV — 2, < £ < N — 2. 
Here X, Y, Z, W are defined by 



X(t) 



r \u 



Y(t) 



r \v 



u v 
with t = lnr, and system (M) becomes 



Z(t) 



W(t) 



r l+b u fi 



X t = X[X-(N-2) + Z], 
, Y t = Y[Y-(N-2) + W], 
{MH){ Zt = z[N + a _ 6Y - Z ], 
W t = W[N + b- fiX -W] 

This system has a Pohozaev type function, well known at least in the case a = b = 0, given at (|l,7p : 



£ H (r) = r 



N 



u'v' + r 



■u 



+ r u 



\v\ S+1 N + avu' N + buv' 



+ 



+ 



N-2 

r uv 



XY 



(J. + 1 5 + 1 5 + 1 r n+l r 
Y(N + b-W) (N + a-Z)X~ 



5 + 1 



■.x- 2 -Y-Z(ZX)^ +i y D (WY) 



(8+l)/D 



XY 



Y(N + b-W) (N + a-Z)X 



<5 + l 



It can also be found by a direct computation, and £h satisfies 

i . . N i . . ( N + a N + b , A 
^ (r) = r «- 1 „V( 7Tr + — -(iV-2)j. 

We define the critical case as the case where (5, //) lie on the hyperbola given by 
N + a N + b 



5+1 n+l 



N-2, equivalents 7 + £ = N - 2. 



(6.1) 



In this case 7 = £ = TfT> an d = 0. It corresponds to the existence of a first integral of 



system (M), which can also be expressed in the variables U= r 7 -u,V= r^-u of Remark [27 

5 H (r) = \J t y t - 7CUV+— - + — — = C. 

/x + 1 5 + 1 

The supercritical case is defined as the case where (6, fi) is above %, equivalently 7 + £ < iV — 2 
and the subcritical case corresponds to (5, fj,) under H. 

Remark 6.1 The energy £ho of the particular solution associated to Mq is always negative, given 
by £h,o = - ^^^ ^^-^Aoyol^oAo)^ 1 )^!^^)^^^- 
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Remark 6.2 In the case a = b = 0, it is known that there exists a solution of the Dirichlet problem 
in any bounded regular domain O ofW N , see for example |i5j/ . \20j; for general a, b, some restrictions 
on the coefficients appear, see f23\j and flffi . 



Next consider the critical and supercritical cases. When a = b = 0, there exists no solution if 
£1 is starshaped, see [36J. Here we show the existence of G.S. for general a, b. The existence in the 
critical case with a = b = was first obtained in [22], then in the supercritical case in [29], and 
uniqueness was proved in [20], [29]. The proofs of [29] are quite long due to regularity problems, 
when 5 or u < 1, which play no role in our quadratic system. 

Remark 6.3 The particular case 5 = jx and a = b is easy to treat. Indeed in that case u = v 
is a solution of the scalar equation Au + |:r| a u = 0, for which the critical case is given by 

5 = (N + 2 + 2a) /(N — 2). Moreover if system (SH) admits a G.S., or a solution of the Dirichlet 
problem in a ball, it satisfies u = V, from [3]. Then we are completely reduced to the scalar case. In 
particular, in the critical case, the G.S. are given explicitely by: u = v = c(K + r ( 2 + a ))(2-A0/(2+a) ^ 
where K = c s ~ 1 /(N + a)(N — 2); in other words they satisfy 113. 3\) with X = Y and Z = W, i.e. 

JV-2 N + a 

Near oo, the G.S. is (obviously) symmetrical: it joins the points Nq and Aq. 

Remark 6.4 Consider the case 5 = 1, a = b = 0, which is the case of the biharmonic equation 

A 2 u = u». 

Recall that it is the only case where the conjecture M.3\) was completely proved by Lin in JJSF- In 
the critical case fi = (N + 4)/(A r — 4), the G.S. are also given explicitely, see [20]: 



u 



(r) = c(K + r 2)(4-A0/2 j K = c ^-i/(iv - 4)(iV - 2)N(N + 2). 



They satisfy the relation XY = N 2 z X + ^r(N — W)Y, and moreover we find that they are on an 
hyperplane, of equation 

(N-2)X(t) Z(t) 
N(N - 4) N 

Observe also that the G.S. is not symmetrical near oo: u behaves like r 4 ~ N and v behaves like 
r 2-iv_ y/ie trajectory in the phase space joins the points Nq and Qo = (N — 4, TV" — 2, 2, 0). 

Proof of Theorem 11.41 1) Existence or nonexistence results: 

• In the supercritical or critical case we apply any of the two conditions of Theorem ll.lt Here 
£ff(0) = 0, and En is nonincr easing; there does not exist solutions of (M) such that at some time 
T, X(T) = Y(T) = N - 2, because at the time T, 



Y(N + b-W) {N + a-Z)X _ 

H+l 5+1 { ' 



N + a N + b W Z 
N-2-- — - + - + 



5+1 n+1 n+1 5+1 



> 
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since W > 0, Z > 0, thus £#(e T ) > 0, which is impossible. Otherwise there exists no solution of the 
Dirichlet problem in a ball B(0,R), because 

£h{R) = R N u'(R)v'(R) > from the Hopf Lemma. 
Then there exists a G.S. The uniqueness is proved in |20j. 

• In the subcritical case there is no radial G.S.: it would satisfy £#(0) = 0, and En is non- 
decreasing, £ff(r) ^ (7 r N-2-7-£ f rom (j5 ip _ an d ry _|_ £ > (_/y — 2), then lim^oo £f/(r) = 0. From 
Theorem II. H there exists a solution of the Dirichlet problem. 

2) Behaviour of the G.S. in the critical case. 

It is easy to see that the condition (|1.6p implies fj. > and 5 > 7773, and that 5 ^ -K^S 
and /i < cannot hold simultaneously. One can suppose that 5 > -j^rf- Let T be the unique 
trajectory of the G.S.. Then £#(0) = 0, thus T lies on the variety V of energy 0, defined by 

XjN + a-Z) Y(N + b-W) = 

5 + 1 n+1 v ' 

From (|5.2p T starts from the point iVo, and from (|5.ip 7~ stays in 

A = {(X,Y,Z,W) G M 4 : < X < N-2, 0<Y<N-2, < Z < N + a, < W < N + b} . 

(i) Suppose that T converges to a fixed point of the system in 7Z. Then the only possible points 

are Ao,Po,Qo which are effectively on V. Indeed Io, Jo, Go, Ho g" V. But Qo = ((N — 2)5 — (2 + 
a), N — 2, N + a — (N — 2)5, 0) K, since 5 > $±§. And P G £ if and only if /i ^ $±§. 

If M > 77ii§ j then T converges to ^4o- If /•* < tt=§7 n o trajectory converges to Aq, from Proposition 
14.51 thus T converges to P - If M / #=§ the convergence is exponential, thus the behaviour of 
u, 1? follows. If /j, = 77^|, then T converges converges to ^4o, = Po', the eigenvalues given by (|10.3p 
satisfy Ai = A2 = N — 2, A3 = N + a — 5(N — 2) < and A4 = 0; the projection of the trajectory 
on the hyperplane Y = N — 2 satisfies the system 

X t = X [X - (N - 2) + Z) , Z t = Z[N + a- 5(N -2) - Z) 

which presents a saddle point at (N — 2,0), thus the convergence of X and Z is exponential, in 
particular we deduce the behaviour of u. The trajectory enters by the central variety of dimension 
1, and by computation we deduce that Y — (N — 2) = —t~ l + 0(t~ 2+£ ) near cxd, and the behaviour 
of v follows. 

(ii) Let us show that T converges to a fixed point. We eliminate W from (|6.2p and we get a still 
quadratic system in (X, Y, Z) : 

X t = X [X - (N - 2) + Z] , 
Y t = Y [Y + b + 2 - (/x + 1)X] + ^jX(N + a-Z), (6.3) 
Z t = Z [N + a - 5Y - Z) . 

We have Xt — 0, and Yt ^ near —00. Suppose that X has a maximum at to followed by a minimum 
at t\. At these times X tt = XZ t , thus we find Z t {to) < < Z t {t\). There exists t 2 G (io? *i) such 
that Zt{t2) = 0, and t 2 is a minimum. At this time Z(t 2 ) = N + a — 8Y(t 2 ), Zttiti) = —5(ZYt)(t 2 ) 
hence 



Y t (t 2 ) = Y(t 2 ) 



Y(t 2 ) + b + 2-^±±X(t 2 

+ 1 



< 
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and X t (t 2 ) < 0, hence (X + Z)(t 2 ) < N - 2, and 



N — 2 — X(t 2 ) > Z(t 2 ) >N + a- 5(^-^X(t 2 ) - b - 2) 

o + l 

(a + 2) + S[b + 2) < - 1)x(t2) = {W + _% + + % x itl) 

but X(t 2 ) < X(to) < 5(N — 2) — (2 + a), which is contradictory. Then X has at most one extremum, 
which is a maximum, and then it has a limit in (0, N — 2] at oo. In the same way, by symmetry, Y 
has at most one extremum, which is a maximum, and has a limit in (0, N — 2] at oo. Then Z has 
at most one extremum, which is a minimum. Indeed at the points where Z± = 0, —Zu has the sign 
of Yf. Thus Z has a limit in [0, iV + a), similarly W has a limit in [0, N + 6) . ■ 

Open problems: 1) For the case 5 = fj,, in the critical case it is well known that there exist 
solutions (u,v) of system (SH) of the form (u,u), such that r" ( u is periodic in t = lnr. They 
correspond to a periodic trajectory for the scalar system (M sca i) with p = 2, and it admits an 
infinity of such trajectories. If S 7^ does there exist solutions (u, u) such that (r 7 n, r^v) is 
periodic in £, in other words a periodic trajectory for system (MH)? 

2) In the supercritical case, we cannot prove that the regular trajectory T converges to Mq, 
that means lim r r^u = A, Mviir^^r^v = B. Here £h{0) = 0, £h is nonincreasing, then Sjj is 
negative. The only fixed points of negative energy are Mq, Go, Ho, but a G.S. satisfies (|5.5p . then 
it tends to (0,0) at 00, hence T cannot converge to Go or Ho from Proposition 14.9} but we cannot 
prove that T converges to some fixed point. 



7 A nonvariational system 

Here we consider system (S) with p = q = 2, a = b and s = m ^ 0. 

(SN) 

where D = 5fi — (1 — s) 2 > 0. In order to prove Theorem we can reduce the system to the case 



-Au 
-Av - 



I 1 a s S 
\X\ U V , 

x\ u^v , 



a = 0, by changing TV into TV" 
Here 



2(N+a) ^ £ rom R emar k 12.41 thus we assume a = in this Section. 



X 



ru 
u 



Y 



2+a 

rv' 
v 



Z(t) 



ru v 



it' 



W(t) 



ru^v 



and system (M) becomes 



(MN) < 



X t = X [X - (N - 2) + Z] , 
Y t = Y [Y - (N - 2) + W] , 
Z t = Z[N - sX -5Y - Z], 
W t = W [N - fiX - sY - W] . 



We have chosen this system because it is not variational, and different hyperbolas in the plane 
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the hyperbola T~L S for which the linearized system at Mq has two imaginary roots, given by 



5 + 1- s fi + l-a N-(N-2)s 

whenever s < 7^2, and 5 + 1 — s > 0, ^ + 1 — s > 0, from Proposition 14. 3| 

• the hyperbola Tio defined by 

<«»» ^T + ^ = V ; <"> 

it was shown in |26j that above %q there exists no solution of the Dirichlet problem; 

• an hyperbola Z s introduced in [38J in case s < , and min(<5, fj) > \s — 1| : 

(Z) 1 1 1 - N ~ 2 (72 ) 

[ * s) 5 + l + + N-(N-2y [LZ) 

• we introduce the new curve C s defined for any s > by 

N N »r (N-2)s . 1 1 - 

(Cs) + -z — - = N - 2 + ^ —L- min( 



5 + 1 2 > + i'<5 + r' 

We first extend and complete the results of |3S] and |26| : 

Proposition 7.1 (i) Assume s < 7^2 , and S + 1 — s > 0, /i + 1 — s > 0. Under the hyperbola Z s , 
system (SN) admits no G.S., and then there is a solution of the Dirichlet problem in a ball. 

(ii) Above T-Lq there exists no solution of the Dirichlet problem. Thus there exists a G.S. 
Proof, (i) We consider an energy function with parameters a, f3,a,9 : 

/ / 1 it+l S 1 5+1 S , °~ I , Q I /rr n\ 

u v + v + pv u H — vu + -uv (7.3) 

r r 

= r N - 2 uv9 = r N - 2 -^(ZXfl 2 (WZylH , (7.4) 

from (|4.2p . where 



£ N (r)=r N 



V (X, Y, Z, W)=XY + aWY + (3ZX - aX - 6Y. (7.5) 

We get 

r 1 - N (uv)- 1 £' N (r) = (a + 9 — (N — 2))XY + (Na - 9)YW + (Nf3 - a)XZ 

- (a(/i + 1) - 1)XY W - (0(6 + 1) - 1)XYZ - asY 2 W - fisX 2 Z. 

Taking a = — , /3 = , we find 

r 3 - N (uv)~ 1 £' N (r) = (a + 6-{N - 2))XY + (Na -9- asY)YW + (N f3 -a- psX)XZ. (7.6) 
If there exists a G.S., from f)5. 1 1) it satisfies X, Y < N — 2, hence 

r 3-JV (w) -1 ^(r) > (a+9-(N-2))XY+((N-(N-2)s)a-9)YW+((N-(N-2)s)P-a)XZ. (7.7) 
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Taking 6 = — ^ 1 2 ^ s , a = — ^ 2 ^ s , we deduce that £' N > under Z s . Moreover Z s is under H s , 
thus 7 + £ > N — 2. Then £n(t) = 0(r N ~ 2 ~' y ~^) tends to at oo, which is contradictory. 

(ii) Taking a = — rj = j?,/3 = -A^ = jj, it comes from (|7.6p 



r 3 - N {uv)- 1 e' N {r) = + ~( N ~ 2 )) XY ~ asY 2 W - fisX 2 Z 

hence £' N < when (IT. 1|) holds. At the value -R where = v(R) = 0, we find £n(R) = 

R N u'{R)v'{R) > 0, which is a contradiction. ■ 

Remark 7.2 (i)When the four curves are simultaneously defined, they are in the following order, 
from below to above: Z S) T-L s ,C s ,T-Lq. They intersect the diagonal 5 = fi repectively for 

. N + 2 n . N + 2 . iV + 2 s . iV + 2 

o = 2s, o = s, o = , o = . 

N-2 ' iV-2 iV-2 2' iV-2 

(m^ For 5 = fi, system (SN) has a G.S. for 5 =| — s. Indeed it admits solutions of the form 
(U,U), where U is a solution of equation —AU = U s+S . Suppose moreover s ^ 5. If 1 — s < 6 < 
-j^l — s, then there exists no G.S; indeed all such solutions satisfy u = v, from [3, Remark 3.3]. 

Then the point P s = ^ -j^rf — s, — s^j appears to be the separation point on the diagonal; notice 
that P S £H S . 



Next we prove our main existence result of existence of a G.S. valid without restrictions on s. 
The main idea is to introduce a new energy function by adding two terms in X 2 and Y 2 to the 
energy £n defined at (|7.3p . It is constructed in order that <£' does not contain Y and Z. Then 
we consider the set of couples (X, Y) such that has a sign, which is bounded by a cubic curve. 
When (5,/j.) is above C s , the cubic curve is exterior to the square 

K = [0, N - 2] x [0, N - 2] , (7.8) 

and then we can apply Theorem 11.11 

Proof of Theorem 11.51 From Theorem II. 1\ if s ^ an the regular solutions are G.S.. 



Thus we can assume s < 



N+2 
N-2' 



Let j, k G M be parameters, and 



<3?(r) = £jv(r) + r 



A? 



S TO 



/2 



s tit; 



'21 



2 m J 2 v 

uV + au^ +1 v s + (3v s+1 u s + -W + -m/ + k- 

r r " 

~ N - 2 -~<-Hzxf 2 (wYy/ 2 *, 



r N 2 uv^ 



ll 11 
S VU .SUV 

2 u' +J 2~ 



where 



9(X, Y, Z, W) = XY + aWY + - aX - 6Y + A:-X 2 + j-Y 2 
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Then 

r^iuv)' 1 ®'^) = (a + 6 - {N - 2))XY + {Na - 9)YW + {Nf3 - a)XZ 

- (a(/z + 1) - 1)XYW - (/3(<5 + 1) - l)XYZ + (j - a)sY 2 W + (k — f3)sX 2 Z 
+ ksX 2 [X — (N — 2)] + jsY 2 [Y - (N - 2)] + (N - 2 - X - Y)(k^X 2 + j^Y 2 ). 

We eliminate the terms in Z, W by taking j = a = — qij, k = (3 = j^j, 9 = Na, a = N(3. Then we 
get the function <3? defined at (|1.9p . Computing its derivative, we obtain after reduction 



B(X,Y) := --r 3 - N (uv)- l &(r) 
s 



f3X 2 {N -2-X) + aY 2 (N -2-Y) + XY 



(3X + aY + -(N - 2 - Na - N/3) 



From Proposition 17.11 we can assume that N(a + (3) — (N — 2) > 0. We determine the sign of 
B on the boundary dK of the square K defined at CLE}. We have B(0, Y) = aY 2 (N - 2 - Y) ^ 
and B(X,0) = (3X 2 (N - 2 - X) ^ 0. In particular £(0,0) = 0. Otherwise B(N - 2, Y) = Y6(Y) 
with 

9(Y) = aY [2(N - 2) - Y] + (N - 2)((N - 2)0 + -(N -2-Na- N/3). 
' s 

On the interval [0,N — 2] , there holds O(Y) > O(0). By hypothesis, (5, fj) is above C s , or equiva- 
lently 

TV s 

(a + /3) iV^2 _1 = 2 min(a,/3); (7 ' 9) 

consequently B(N — 2, Y) ^ and similarly B(X, N — 2) ^ 0. Then B is nonnegative on dK and is 
zero at (0, 0), (0, N — 2), (N - 2, 0). The curve B(X, Y) = is a cubic with a double point at (0, 0), 
which is isolated under the condition (|7.9p : B(X, Y) > near (0,0), except at this point. Then 
B(X, Y) > on the interior of K. 

Suppose that there exists a regular solution such that X(T) = Y(T) = N — 2 at the same time 
T. Indeed up to this time (X, Y) stays in K, thus the function $ is decreasing. We have $(0) = 0, 
and at the value R = e T , we find 



= R N - 2 -i-*(N -2] 



aIY + /3Z , /V 

1 - (0 + £*)(■ 



2 2 ; 



iV-2 v ' V JV 

then &(R) > 0, since min(a,/3) < a + /3. Therefore from Theorem II .11 there exists a G.S. 



Remark 7.3 We wonder if the limit curve for existence of G.S. would be 7i s , or another curve C s 
defined by 

1 1 N-2 

( s) ^TT + ^TT ~ N (n-2)s ' 

which ensures that &(R) > 0, and also B(N — 2, N — 2) > 0. 77ns curve cuts the diagonal at the 
same point P s = ^-^r§ — s, jfz 2 ; — s^j as 7i s . Notice that C s is under ~H S . 
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8 The radial potential system 



Here we study the nonnegative radial solutions of system (SP) : 

-A p u = \x\ a u s v m+1 , 



(SP) 



-A q v = \x\ a u s+1 v m , 



with a = b, 6 = m + l, fj, = s + l, and we assume f)1.5|) . System (M) becomes 



(MP) < 



X t = X 
Y t = Y 



p—i 1 p—i 
Y - ^2 + 



9-1 ' 9-1 

Z t = Z [N + a - sX - (m + 1)V - Z] , 
W t = W [N + b - (s + 1)X - mY - W] . 

For this system D, 7 and £ are defined by 

D = p(l + m) + q(l + s) —pq, (p—l — s)j+p + a=(m + l)t;, (q- 1 - m)^ + q + b = (s + 1)7, 
thus 7 and £ are linked independtly of s, m by the relation 

pg(m + s + 2 + a) 



p( 7 + 1) = g(£ + 1) 



(8.1) 



The system is variational. It admits an energy function, given at (|1.13p . which can also can be 
obtained by a direct computation in terms of X, Y, Z, W: 



£ P (r) = i> 



S ~\~ 1 7TI ~4~ 1 

ZW W((N -p)-{p- l)X) Z((N -q)-{q- l)Y) 



U) 



where 



r N-2-a 1 np-1 yiq-l 



„AT-( 7 +l)p 



Then we find 



^-5(a+l)(p-l)y-p(m+l)(g-l)^p(g-m-l)^yg(p-s-l) 



1/fl 



5'(r) = (JV + a — (s + 1)^— ? - (m + 1)- «) r tf- 1 +«V+ 1 t ,'"+ 1 

Thus we define a critical line P as the set of (<5, //) = (m + 1, s + 1) such that 

N + a = (m + 1)^^ + (a + 1) 

9 P 

equivalent to pq(m + s + 2 + a) = iVZ), or N + a = (m + 1)£ + (s + 1)7, or 

,iV-p iV-g, 



.3) 



(7,0 = ( : 



P 



TV-p t ^ N-g 

(s + 1) 7 + (m + 1)£ > iV + a. The supercritical case is the set of points above V. 



The subcritical case is given by the set of points under V, equivalently 7 > —j^, £, > or 
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Remark 8.1 The energy (<?p)g of the particular solution associated to Mq is still negative: (£p)q 

l/D 



D_ r N+a-{-y+l)p 



pq 



j^9(p- 1 )yP(9- 1 )^9( s + 1 )^P( m + 1 ) 



Remark 8.2 When p = q = 2, another energy function can be associated to the transformation 
given at Remark \2.2[ the system \2. 9\) relative to u{r) = r~ 7 £/(i), v(r) = r~*V(t) is 



U tt + (N - 2 - 2 7 )U t - 7(iV - 2 - 7 )U + U s V m+1 = 
V tt + (N - 2 - 2 7 )V t - j(N - 2 - 7)V + U s+1 V m = 



and the function 



E P (t) = i±i(U? - 7 (JV - 2 - 7 )U 2 ) + ^(V 2 - 7 (JV - 2 - 7 )V 2 + u^ 1 ^ 1 



satisfies 



(E P ) t = —(N — 2 — 2 7 ) [( S + 1)U 2 + (m + l)V t 2 ] 
differs from Sp, even in the critical case. This point is crucial for Section^ 



(8.4) 



.5) 



It has been proved in |34| . [35], that in the subcritical case with a = 0, there exists a solution of 
the Dirichlet problem in any bounded regular domain Q, of W N ; and in the supercritical case there 
exists no solution if £1 is starshaped. Here we prove two results of existence or nonexistence of G.S. 
which seem to be new: 



Proof of Theorem II. 61 1) Existence or nonexistence results. 

• In the supercritical or critical case there exists a G.S. From Theorem 1 if it were not true, 
then there would exist regular positive solutions of (MP) such that X(T) = and Y(T) = ^Ex- 
it would satify £p ^ 0. Then at time T, we find £p(R) > 0, from (I8.2p . since W > 0, Z > 0, which 
is impossible. 

• In the subcritical case, there exists no G.S. Suppose that there exists one. Now £p is nonde- 
creasing, hence £p ^ 0. Its trajectory stays in the box A defined by (|5.1|) . thus it is bounded. If 
q ^ m + 1 and p ^ s + 1, we deduce that , £p{r) = 0(r N ~^ +1 ^ p ) from (|8.2p . then £p tends to at 
oo, which is contradictory. Next consider the general case. We have 



£p(r) < r 7V ~( 7+1 ) p X q ( p ~ lS> Y p ( q ~ lS> Z p ( q ~ m ~ lS> W q ( p ~ s ~ lSl 
r N-(j+l)p ^c2(p-l)yp(<?-l)^-(?(l+s) jyp(l+m)1 1 



l/D 



ZW 



then the same result holds. Consequently, from Theorem 1 there exists a solution of the Dirichlet 
problem 

2) Behaviour of the G.S. in the critical case. 

Let T be the trajectory of a G.S.; then £p(0) = 0, thus T lies on the variety V of energy 0, also 
defined by 



qW [{s + l)((p - 1)X — (N - p)) + pZ] = p(m + \)Z \{N 



1.6) 
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and Y < ^f, hence (s + l)((p - 1)X — (N — p)) + pZ > 0. From ([B3]) , T starts from _/V = 
(0,0, JV + a, N + b) and stays in .A. Eliminating W in system (M), we find a system of three 
equations 

X t = X \x - £=Z + -ZJ , 
1 p— l p— l ' 

Y t = YF, 

k Z t = Z [iV + a - sX - (m + 1)Y - Z] , 



where 



F{X,Y,Z) 



N 



q-l 



Y 



p(m + 1 - q)Z + g(s + 1)((JV - p) - (p - l)X) 
{s + l){{p-l)X -{N -p))+pZ 



(i) If T converges to a fixed point of the system in TZ, the possible points on V are Aq,Iq, Jo, Pq, 
Qo,Go, Hq, Rq,Sq. The eigenvalues of the linearized problem at Aq, given by (|10.3|) satisfy 

A 1; A 2 > 0, A 3 = N + a - s^^- - (m + 1)^^ ^ A 4 = A* = N + a - (s + 1)^^ - m N ~ '' 



p — 1 



9-1 



p — 1 



9-1 



since q S Pi and A3 < A* for q 7^ p, and A3 = A* < for q = p, from (|8.3p . Then Aq can be attained 
only when A* 5^ 0, from Proposition 14.51 And Pq can be attained only if 



q>m + l, A* ^ and q + a < (s + 1 



N -p 

7" 5 
P — 1 



(8.7) 



from Proposition 14.61 because 7 = < ^r^- We observe that the condition A* ^ joint to (18.3 
implies m + 1 < q < p and is equivalent to (|8.7j) . Indeed it implies 



N - v N 

-(s + 1) S N + a - m— 

p — 1 



iV + a 



then 



g-1 q-l 
N — p q — p 



(iV + (1 _^_ (s + 1) w -f- 



+ 



< 



thus q < p. From (|8.3|) we obtain 

. ,m + 1 
(tf- g (__-!) = , 



p — 1 p 

N — p 



a + q), 



+ a - (s + 1 



P 



^ / , X — p ,p — q 

S(s + l) ^ (£—7-1 <0 ' 

p p — 1 



hence m+1 < q and (|8.7p follows. By symmetry, Qq cannot be attained since q ^ p. Then ^4o an d 
Po are incompatible, unless Aq = Pq, and Po is n °t attained when p = q. 

(ii) Next we show that T converges to A§ or to Pq. If t is an extremum value of Y, then 

(— 1_ - l)Z(t) + ^—{{N-p)-{p- l)X(i)) = 0. (8.8) 



This relation implies q > m + 1 and 



p — 1 



1 + 



p(ra + 1 



9(5 + 1) 



DX(t)Z(t) 
(p-l)q(s + l) 



> 0. 
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In the same way, if t is an extremum value of X, then p > s + 1 and Yt(t) > 0. Near — oo, there 
holds Xt,Yf ^ 0, and Zt,Wt — 0, from the linearization near Nq. Suppose that X has a maximum 
at ^ followed by a minimum at t\. Then p > s + 1, and Y is increasing on [to,ti]- At time to we 
have (p — l)X(to) + Z(to) = N — p and X u {to) S 0, thus Z t (to) ^ 0; eliminating Z we deduce 
p + a + {p-l - s)X(t ) ^ (m + l)y(to) and similarly (m + l)F(ti) ^p + a + (p-l- s)Af(ii); 
hence V(ti) < Y(io), which is a contradiction. Thus X and Y can have at most one maximum, and 
in turn they have no maximum point. Therefore X and Y are increasing, and they are bounded, 



hence X has a limit in 

at each time where Zt 
is impossible. 



N-p 



J ' p-l 

0, we have Z u 



and Y has a limit in ( 0, 



N- 



9-1 



Then Z, W are decreasing; indeed 
Z{— sXt — (m + 1)Y) < 0, thus it is a maximum, which 



Then T converges to a fixed point of the system. Moreover, since X and Y are increasing, it 
cannot be one of the points Jo, Jo, Go, Hq, Rq, So. It is necessarily Aq or Pq. We distinguish two 
cases: 

• Case q ^ m + 1. Then T converges to ^4o, and A3, A* < 0, then (jl.lOp follows. 

• Case q > m + 1. Then T converges to ylo (resp. Po) when A* ^ (resp. A* ^ 0). If the 
inequalities are strict, we deduce the convergence of u and v from Propositions 14.51 and 14. 6} and 
(jl.lip follows. If A* = 0, then Po = Ao, and A3 = ^zjyfe^ < 0- The projection of the trajectory 

T in M 3 on the plane Y = satisfies the system 



X f = X 



X 



+ 



z 



N-p 

p — 1 p — 1 



iV + a - - (m + 1) 



N 



Z 



which presents a saddle point at ( ^_f , 0), thus the convergence of AT and Z is exponential, in 
particular we deduce the behaviour of u. The trajectory enters by the central variety of dimension 



1, and by computation we deduce that Y 



N- 



1 



q—1 q — l—m 



t' 1 +0{t- 2+£ ), then (TLT2D follows. 



9 The nonradial potential system of Laplacians 

Here we study the possibly nonradial solutions of the system of the preceeding Section when 
p = q = 2 : 

(SL) 



-Au = \x\ a u 8 v m+1 , 



-Av 



x 



u s+l v m 



with D = s + m. We solve an open problem of [7J: the nonexistence of (radial or nonradial) G.S. 
under condition fjl . 141) . 

It was shown in [7] in the case N + a ^ 4. The problem was open when N + a < 4, and 
m + s + 1 > (N + a)/(N -2), which implies N < 6. Indeed in the case m + s + l < (N + a)/(N -2), 
there are no solutions of the exterior problem, see [6} Theorem 5.3]. Recall that the main result of 
[7] is the obtention of apriori estimates near or 00, by using the Bernstein technique introduced 
in [18] and improved in [8]. Then the behaviour of the solutions is obtained by using the change of 
unknown 



u(r, 9) = r~^\J(t, 6), v(r, 6) = r^Y(t, 9), 



t = In r, 
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extending the transformation of Remark 18.21 to the nonradial case (in fact here t is —t in [7J); it 
leads to the system 

U tt + (JV - 2 - 2 7 )U t + A S U - 7 (iV - 2 - 7 )U + U s V m+1 = 0, 
V tt + (N - 2 - 2 7 )V t + A 5 V - 7 (iV - 2 - 7 )V + U s+1 V m = 0, 

where As is the Laplace-Beltrami operator on Sn-i- A corresponding energy is introduced in [TJ: 



E L (t) 



s + 1 



+ 



2 

S N 

m + 1 



(U t 2 - |V 5 U| 2 - 7 (iV - 2 - 7 )U 2 )d# 
/" (V 2 -|V s V| 2 -£(iV-2-£)V 2 )^+ y U s+1 V m+1 d#, 

extending (|8.5p to the nonradial case; it satisfies 

(E L ) t = -(N -2-2 7 ) J [{a + 1)U 2 + (m + 1)V 2 ] ^ 

5 jv-i 

Here we construct another energy function, extending the Pohozaev function defined at (|1.13p to 
the nonradial case. 



Lemma 9.1 Consider the function £z,(r) defined by 



r- N £ L (r) 



+ 



s + 1 
2 

S N 

m + 1 



u 2 r -r- 2 \V s u\ 2 + {N-2)^ 

r 



de 



(^) 2 -r- 2 |V 5 ,| 2 + (iV-2S — 



vv r 
r 



S N-1 

Then the following relation holds: 



d6 + r a J u s+1 v rn+l d9. 

S N-1 



AT _ O AT _ 9 

r 1 - N £' L {r) = {N + a-{s + l)— ( m + i)__ ) r a \ u s+l v m+1 d0. 



S N-1 



Proof. In terms of t, we find 

£ L (t) = £ L ,i(t) + e Lj2 (t) + s Li3 (t), with 



£L,l(t) 



S + 1 



(N-2)t 



u\ - |V 5 -u| 2 + {N- 2)uu t 



d6, 



S N-1 



m+1 



{N-2)t 



\V s v\ 2 + {N -2)vv t ]d6, £ L3 (t) = e {N+a)t / u s+1 v m+1 d9 



S N-1 

and u satisfies the equations 



S N- 



Utt 



+ (N — 2)u t + A S U + e (2+a)* u ^m+l = Q) 



(e^'ut)* + e^" 2 )* A s u + e^ 



t u s v m+l 



0. 



(9.1) 
(9.2) 
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and v satisfies symmetrical equations. Multiplying (|9.2p by u and (|9.ip by (s + l)e^ N 2 ^ut, we 
obtain 

0= / uCe^-^uOt + e^" 2 )* / uAsu+eW / u s+1 V m+1 



5JV-1 S N_1 S N ~ 1 

--j t J ueW-Wut-eW- 2 * J ( U 2 t +\V S u\ 2 ) + e^ N +^ | 

gN-l S N_1 S N_1 



u s+1 v m+1 



1 J ^(N-2)ue^K t - S -±^(N-2)e^ J (u 2 + \V s u\ 2 ) 

5JV-1 s N ~ 1 

8 + 1. 



-(N-2)e( N+a » J u s+1 v m+ \ 

gN-l 

and symmetrically for t>, and adding the equalities we deduce 

,2 _ IY7„„,I 2 



0= ( S + l)( e (^| I ("t-IVg"! + (JV _ 2)e (AT-2)t I 



2 



gN—1 S 1 ^ -1 
)'l [ V t-\ V Sv\\ (N _ 9U N-2)t I 2 



d_ 



(N-2)t 



+ (m + l)(e^-^- y * 2 a ' +(iV-2)e 

gN-l S N ~ 1 
+ ^ e (N+a)t J n s+l w m+l ) _ (iv + a)e (7V + a) i ^ ^+1^+1 

gN-l S N_1 



j ({s + l)(u 2 t -\V s u\ 2 ) + (m + l)(v 2 t -\Vsv\ 2 ))+e^ N+a ^ j u s+l v' 



2 

gN-l S N ~ X 

/V — 2 f 

+ -3- e (7V - 2) * y ((a + 1)(" 2 + |VH 2 ) + (m + l)(t> 2 + |V s t;| 2 )) 

S N-1 

= (N + a)e( N+a * J u s+l v m+ \ 

S N-1 

hence 

N — 2 N — 2 f 

(£ L ) t ( t ) = (N + a - (s + 1)— { m + l)——)^ N+a ) t j u s+1 v m+l d9. 

gN-l 

m 

Proof of Theorem [TT71 Suppose that there exists a G.S. Since s+m+1 < (N+2+2a)/(N -2) 
we deduce that El and El are increasing and start from 0, then they stay positive. From [7j 
Corollary 6.4] , since s + m + 1 < (N + 2)/ (N — 2) , three eventualities can hold. The first one is that 
(u,v) behaves like the particular solution (uo,vq); it cannot hold because El has a negative limit, 
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see Remark 6.3]. The second one is that (u,v) is regular at oo, that means limui^oo \x\ 2 u = 
a > 0, limi a .i_ H j Q Ix^ -2 ?; = ft > 0; it cannot hold because lim^ 00 £'i(t) = 0. It remains a third 
eventuality: when for example m > (JV + a) /(JV — 2), and (u, v) has the following behaviour at oo : 

lim u = a > 0, and lim \x\ k v = j3 > or 0, with k = (2 + a)/(m — 1). (9-3) 

r->oo |a;|-K» 

The condition on m implies that JV < A— a from assumption (|1 . 14[) . In that case lim^oo Ei{t) = oo, 
which gives no contradiction. Here we show that a contradiction holds by using the new energy 
function 

First recall the proof of (|9.3p . Making the substitution 

u(r,0) =u(t,9), v{r,9) =r- k V{t,6), t = lnr, 6 <E Sjv-i, 

we get 

utt + (N- 2)ut + A s u +e -^t u s V m+i = Q) 
V tt + (JV - 2 - 2/¥)V t + A S V - k(N - 2 - k)V+u s+1 V m = 0. 1 j 

Then u, V are bounded near oo, and from [TJ Proposition 4.1] u converges exponentially to the 
constant a, more precisely 

|||w-a| + \u t \ + |V5w|||co(^-i) = 0(e" {Ar_2)t ), (9.5) 

because k ^ (JV — 2)/2 and all the derivatives of V up to the order 2 are bounded. The equation 
in V takes the form 

V tt + (JV - 2 - 2k)V t + A S V - k(N - 2 - k)V+a s+1 V m + ip = 

where if and its derivatives up to the order 2 are 0(e~( N ~ 2 > t ). From [TJ Theorem 4.1], the function 
V converges to /3 or to in C 2 (S N ~^), 

Next we define 

/(t) = e (^- 2 ) i J Utde = r N - 1 J u r d9. 

gN-l S N_1 

Then 

S L ,i(t) = (JV - 2)i±i«/(t) + 0{{e^ N -^)) 



from (|9.5|) . Moreover from 

f t (t) = - e ^-2-2k)t J u s V m+l de < q 

S N-1 

since u is reg ular at 0, f{t) = O^" 1 )*) at — oo, in particular limt_»._ 00 f(t) = 0. And f t (t) 
0(e^ N ^ 2 ^ 2k ^) = 0(e~*) at oo, then f(t) has a finite negative limit — £ 2 ; and 

li m S Ltl (t) = -(N-2)^±±a£ 2 . 

t— >oo Z 

Moreover u = e _fc *V, and V and its derivatives up to the order 2 are bounded, thus 

e L>2 (t) = 0{e^ N - 2 - 2 ^) = 0(e-*) 

Finally 

S Lj3 (t) = 0{e^ N+a -^ m+ ^ t ) 
and N + a — k(m + 1) < < 0. Then £x has a finite limit # < at oo, which is contradictory. 
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10 Analysis of the fixed points 

Here we make the local analysis around the fixed points. 

Proof of Proposition 14.41 (i) Consider a regular solution (u, v) with initial data (uo,Vq). 
When when r — > 0, we have 

(_r"-i | u '| p - 2 u y = r N - 1+a uU(l + o(l)), - \u'\ p - 2 u' = ^l_r 1+a TiS«g(l + o(l)), 
thus from (|2.ip . when t — > — oo 

X ^ = ( F ^^ +1 - p ^) 1/(P ~ 1) e (p+a)t/(p - 1) (l + 

y W = (^< Vo m+1 - 9 ) 1 /^- 1 )e^/^- 1 )(l + o(l)), 

and limt-^-oo Z = N + a, lim t ^__ 00 = (iV + 6). In particular the trajectory tends to Nq = 
(0,0, iV + a,N + b). 

(ii) Reciprocally, consider a trajectory converging to No. Setting Z = N + a + Z ,W = N + b + W, 
the linearized system is 

X t = ^^X, Y t = ^^-Y, Z t = (N+a) \-sX - SY - z] , W t = (N+b) \-fiX -mY-W 
p-1 q-1 I J L 

(10.1) 

The eigenvalues are 

Xi = P±a >Q ^ A2 = i+6 0; A3 = _ (Ar + a) <0) A 4 = -(A^ + 6) < 0. (10.2) 
p — 1 g — 1 

The unstable variety V u and the stable variety V s have dimension 2. Notice that V s is contained 
in the set X = Y = 0, thus no admissible trajectory converges to iVo when r — > oo, and there 
exists an infinity of admissible trajectories in TZ, converging to Nq when r — > 0. Moreover we get 
lim^_ 00 e-(P +a )/(P- 1 )*X(t) = k > and lim^_ 00 e-fo+Wfo-^Yft) = £ > 0. Thus («,v) have a 
positive limit (it ,u ) = ((N + o)/sP- 1 )(« -1 - m )/ iJ ((JV + b)^~ l ) 5 / D from dOJ, (03]), hence (u,v) is 
a regular solution. 

Next we show that for any n > 0,£ > there exists a unique local solution such that 
lim t ^^ 00 e- ( -P +a ^/ ( - p - 1 ^X(t) = k and lim^_ 00 e -(9+6)/(<?-i)*y = £. On V„, we get a system of 
two equations of the form 

X t = X(A X + F(X, K)), Y t = Y(\ 2 + G(X, Y)), 

where F = + BY + /(X,Y), where / is a smooth function with /x(0,0) = /y(0,0) = 0, 
similarly for G. Setting X = e Xlt (n + x), Y = e x ' 2t (£ + y), and assuming A2 ^ Ai and setting 
p = e Xlt we obtain 

x P = -(« + x)F(p(« + x), /^/Ai (£ + y p = + y)G(p(« + x), / 2 / Al (£ + y)), 
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with x(0) = y(0) = 0. Then we get local existence and uniqueness. Hence for any uq,vq > there 
exists a regular solution (u, v) with initial data (uq,vq). Moreover u, v € C 1 ([0, R)) when a,b > — 1. 



Proof of Proposition 14. 5L The linearization at Aq 



N-p 



p — 1 
The eigenvalues are 
N-p 



Ai 



p — 1 



>0, A 2 



X + 



N 



Z 



p — 1 



N 



q-1 



Y + 



W 



1 



N - v N 
> 0, A 3 = N+a-s --5- 



p — 1 



1 



f=£ f=f 0,0) gives, withX 



Z = A 3 Z, w t = \ 4 W. 



1 \ ATj_h N ~ P N 

, A4 = iv +0— \x : — m 



p-1 



(10.3) 

• Convergence when r — > 00 : If A3 > 0, or A4 > 0, then the stable variety V s has at most dimension 1, 
it satisfies W = or Z = 0, hence there is no admissible trajectory converging to Aq at 00. If A3 < 0, 
and A4 < 0, then V s has dimension 2. Moreover V s Pi {Z = 0} has dimension 1 : the corresponding 
system in X, Y,W has the eigenvalues Ai,A2,A4; similarly V s PI {W = 0} has dimension 1. Then 
there exist trajectories in V s such that Z > and W > 0, included in 1Z and thus admissible. They 
satisfy lime- A3 *Z = C 3 > 0,lime~ A4 *VF = C 4 > 0, then P~TTj) follows from (]43j) . 



• Convergence when r — > : If A3 < 0, or A4 < 0, the unstable variety V u has at most dimension 
3, and it satisfies W = or Z = 0. Therefore there is no admissible trajectory converging at —00. 
If A3,A4 > 0, then V u has dimension 4; in that case there exist admissible trajectories, and (I4.1ip 
follows as above. ■ 



Proof of Proposition I4.6L We set Po 

^ -(« + &) 



F„0,W*J , with 
(g-l)(JV + 6-|=2 M )-m(JV-g) 



g — 1 — m 

for m + 1 7^ q. The linearization at Pq gives, with X 



q — 1 — m 



^+X,Y = Y, + Y,W = W* + W, 



X f 



N-p 



X + 



z 



p-1 



p — 1 

The eigenvalues are 

N-p 

Ai = f > 0, A 3 

p — 1 

and the roots A2, A4 of equation 



Yt = Y* 



Y + 



II" 



Z t = A 3 Z, W t = W* 



-liX - rnY-W 



N + a-s^-SY = ^— 
p — 1 q — 1 — m 



P — 1 



A 



2 (y« - w*)a + = 



Then if A3 < (resp. A3 > 0) there is no admissible trajectory converging when r — > (resp. 
r -> 00). Indeed V u = V u n {Z = 0} (resp. V s = V s n {Z = 0}). 
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1) Suppose that q > m + 1. Since q + b < ^zrA 4 < N + 6 



m^pYi we have Po G and 
A2A4 < 0. First assume A3 < 0, that means 7 < • Then V s has dimension 2, and V s fl {Z = 0} 
has dimension 1, there exists trajectories with Z > 0, which are admissible, converging when 
r — > 00. Next assume A3 > 0. Then V u has dimension 3, and V u H {Z = 0} has dimension 2. Thus 
there exist admissible trajectories converging when t — > — 00. 

N-u 



2) Suppose that q < m + 1. Since q + b > -^pifi > N + b- 

N-p 



■m- 



zf, we have Po G 72., and A2A4 > 0. 

We assume q^Y^ + m(JV — q) ^ N(q — 1) + (6 + that means Y* 7^ TV*. First suppose A3 > 0, 

that means 7 < N Z% ■ If FteA2 > 0, then V u has dimension 4, or ReA2 < then V u has dimension 
2 and V M fl {Z = 0} has dimension 1. In any case, there exist admissible trajectories converging 
when r — > 0. Next assume A3 < 0. If ReA2 > 0, then V s has dimension 1, and V s fl {Z = 0} = 0. 
If ReA2 < 0, then V s has dimension 3. In any case V s contains trajectories with Z > 0, which are 
admissible, converging when r — > 00. 

Those trajectories satisfy lime _A3 *Z = C3 > 0, limX = ^Ey 1 , limY = Y* and limlY = W*, 
thus (ITOD follows from flOD and ' ■ 

Proof of Proposition US The linearization at I = (^=2. n n n^™, m ;+v, v - ^ 



iV 



P 



z 



p — 1 ' p 
The eigenvalues are 
AT -p 



Y 



N 



-Y, Z t = (N+a-s 



N 



0, 0, 0) gives, with X 



Ai 



P 



1 



>0, A 2 



AT 



< 0, A 3 = N + a 



P 



A^ 



1 



)Z, W t = (i\T+&-/z 



p-i 
N 



+ X, 



P 



P 



)W. 



p 



A 4 = N + b- n- 



N 



P 



p — i p — l 

• Convergence when r -> 00 : If A 3 > or A 4 > 0, then V s = V S (~){Z = 0} or V s = V s n {W = 0}. 
There is no admissible trajectory converging at 00. Next suppose that A3,A 4 < 0. Then V s has 
dimension 3; it contains trajectories with Y,Z,W> 0, which are admissible. They satisfy lim X = 
lime- A2 *Y = C 2 > 0, lime~ X:it Z = C 3 > 0, lime- A4 *IY = C 4 > 0, then (@J5§ follows from 
fal) and (122 



• Convergence when r — > : Since A2 < we have V u = V u fl {Y = 0} , hence there is no admissible 
trajectory converging when r —¥ 0. ■ 

Proof of Proposition H79l The point G = , 0, 0, N+b-^ffi) G 72 since ^ff/x < iV+6. 
The linearization at Go gives, with X — 



z f 



N 



P 



p — 1 

(iV + a 



X + 



Z 



p — 1 



^+X,IY = A^ + 6- 
Y , . iV 



AT 



p, 



The eigenvalues are 
AT-p 



A 



P 



1 



>0, A 2 



1 



1 



p — 1 



(q + b 



Z, 



N 



Yf- 
W t 



q-l 
-■(N + b 



(q + b- 
N 



P 

~P 



p — 1 



P 



//), A 3 = AT + a 



N 



P 



p — 1 ' " p — 1 ' p 

• Convergence when r -> 00 : If A 2 > 0, or A3 > 0, then V s = V s n {Y = 0} or V s = V s n {Z = 0}, 
there is no admissible trajectory converging at 00. Assume A2, A3 < 0, then V s has dimension 3, it 
contains trajectories with Y, Z > 0, which are admissible. 



Ai 



N 



P 



1 



■/' 



A^-6<0. 
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• Convergence when r — > : If A3 < 0, or A2 < there is no admissible trajectory. If A2,A3 > 
then V s has dimension 3, it contains admissible trajectories. 



N-p 



f, lime- A2 *y = C 2 > 0, lime _A3t Z = C 3 > 0, lim W = N + b 



-\ s t> 



In any case limX 
hence (j4T3j) still follows from (g3J) and fl27 

Proof of Proposition RFTTUl We set C = (0, Y, 0, JY) , with 



Y 



q + b 



m + 1 — q ' 



W 



m(N-q) - (N + b)(q-l) 
m + 1 — q 



N-p 



(10.4) 



Then Co € 7£ if > N + b, implying q < m + 1. The linearization at Co gives, with Y = Y + Y 

and W = JY + W 



N-p 



p — 1 
The eigenvalues are 



X, Y t = Y 



Y + 



W 



N-p 
p — 1 



, z t = \ 3 z, w t = w 



, A 3 = N + a-8Y, 



-fjtX -mY-W 



and the roots A2, A4 of equation 



A 2 - (Y - W)A + 



m + 1 — g 
9-1 



riy = 



(10.5) 



then A2A4 > 0. We assume m / — jy^ 6 " 1 " 1 ^ , that means 7 /IV. 

• Convergence when r — > 00 : if A3 > we have V s = V s fl {Z = 0} , hence there is no admissible 
trajectory. Next assume that A3 < 0, that means 5 > (N + a) m t + ^ q -If Re A2 < (resp. > 0) then 
V s has dimension 4 (resp. 2) and V s H {X = 0} and V s n {Z = 0} have dimension 3 (resp. 1) then 
there exist trajectories with X, Z > 0, which are admissible. 



In any case lime _Al< X = C x > 0, limY = Y, lime~ A3 'Z = C 3 > 0, limlY = W, then P~T5|) 
follows. 

• Convergence when r — > : Since Ai < we have V u = V u fl {X = 0} , hence there is no admissible 
trajectory. ■ 

Proof of Proposition 14. Ill We set R = (0, Y, Z, W) , where Y, W are defined at (|10.4[> . and 

Z = N+a-S j^ . Under our assumptions it lies in ft. Setting Y = Y+Y, Z = Z+Z, 1Y = W+W, 
the linearization at Rq gives 



X* = AiX, Y t = Y 
the eigenvalues are 



Y + 



A, 



W 



Z, 



-sX -SY-Z 



(p + a-d — ), A 3 



p — 1 X1 m+1 — q 

and the roots A2 , A4 of equation of equation (|10.5j) . 



W t = W 



Z < 0; 



-iiX -mY-W 
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• Convergence when r — > oo : If Ai > 0, that means (p + a) m ^ + ^ q < <5, then V s = V s PI {X = 0} , 
hence there is no admissible trajectory. Next assume Ai < 0; if Re A2 < (resp. > 0) then V s 
has dimension 4(resp. 2) and V s n {X = 0} has dimension 3 (resp. 1) then there exist admissible 
trajectories. 

• Convergence when r — >■ : If Ai < 0, then V u = V u n {X = 0} , hence there is no admissible 
trajectory. Next assume Ai > 0. If Re A2 = Re A4 < (resp. > 0) then V s has dimension 4 (resp. 
2) and V s H {X = 0} has dimension 3 (resp. 1) then there exist admissible trajectories. 

In any case lime~ Al 'AT = C\ > 0, limy = Y, limZ = Z, limW = W, then (|4.15p holds again.B 



Remark 10.1 Finally there is no admissible trajectory converging to = (0,0,0,0), or Kq 
(0, 0, iV + a, 0), or Lq = (0, 0, 0, N + 6). Indeed the linearization at gives 



X t 



N -p 



X, Y t 



N -q 
4-1 



y, Z t = (iV + a)Z, W t = (Af + 6)W 



Then V s and V u have dimension 2, hence V s is contained in {Z = W = 0} , and V u in {X = Y = 0} . 
The linearization at Kq gives, with Z = N + a + Z , 



x 4 = ^x, y 
p — 1 



iV 



'y, Z t = (iV + a) 



-sAT - - Z 



, Wt = (JV + b)W. 



The eigenvalues are — -^±, —(N + a), N + b. Then V s and V u have dimension 2, hence V s is 
contained in {Z = W = 0}, and V u in {Y = 0} . The case of Lq follows by symmetry. 
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